CARLA

CARLA (Car Learning to Act) is an open-source platform for developing, training and evaluating autonomous driving agents, built on top of Unreal Engine.

North stars

• Win CARLA AD Challenge
• Build something like this end-to-end
• Generate 100,000 images

Resources

• CARLA Workshop by WATonomous
• CARLA Tutorial to get data on remote Ubuntu GPU Server

Important: https://carla.readthedocs.io/en/latest/adv_synchrony_timestep/

watod -t gui_tools
rosrun carla_config scenario_ped_crossing.py

CARLA Synthetic Data Generation

Writing notes about how to use CARLA with a focus on how we use this to generate synthetic data.

Asynchronous vs. Synchronous Mode

• Asynchronous mode (DEFAULT) $\to$ Server runs the simulation as fast as possible, without waiting for the client
• Synchronous mode $\to$ Server waits for a client tick, a “ready to go” message, before updating to the following simulation step.

Variable Time-Step vs. Fixed Time-Step

• Variable time-step (DEFAULT) $\to$ The simulation time that goes by between steps will be the time that the server takes to compute these
• Fixed time-step $\to$ The elapsed time remains constant between steps

Important

In Synchronous mode, always USE Fixed Time-Step.

If you want things to run as fast as possible, use Asynchronous + Fixed Time-Step

• Why not variable time-step? This is like: however long it takes to render the next frame, CARLA will take its time. But with fixed time-step, it’s like an impending deadline, i.e. you only have 0.1 seconds to render this, do whatever you can.

CARLA Recorder

Record https://carla.readthedocs.io/en/latest/adv_recorder/

Actors vs. Blueprints https://carla.readthedocs.io/en/latest/core_actors/

Actors: elements that perform actions within the simulation, and they can affect other actors. Actors in CARLA includes

• vehicles
• walkers
• sensors,
• traffic signs and traffic lights
• the spectator

So basically, everything we spawn in an actor.

Blueprints are layouts that allow the user to smoothly incorporate new actors into the simulation. They are already-made models with animations and a series of attributes.

You can get a particular blueprint from the blueprint library.

But generally, it seems that we just directly choose an actor from the blueprint library, like the following

vehicle_bp = random.choice(blueprint_library.filter('vehicle.*.*'))

Life Cycle:

• Spawning
• Handling
• Destruction

Coordinate System

CARLA uses the Unreal Engine coordinates system. Remember that carla.Rotation constructor is defined as (pitch, yaw, roll), that differs from Unreal Engine Editor (roll, pitch, yaw).

Town

Summary

• Town01: A basic town layout consisting of “T junctions”.

• Town02: Similar to Town01, but smaller.

• Town03: The most complex town, with a 5-lane junction, a roundabout, unevenness, a tunnel, and more.

• Town04: An infinite loop with a highway and a small town.

• Town05: Squared-grid town with cross junctions and a bridge. It has multiple lanes per direction. Useful to perform lane changes.

• Town06: Long highways with many highway entrances and exits. It also has a Michigan left.

• Town07: A rural environment with narrow roads, barns and hardly any traffic lights.

• Town10: A city environment with different environments such as an avenue or promenade, and more realistic textures.

CARLA attachments: You can choose between a rigid attachment, and a spring_arm attachment.

CARLA Sensors: https://carla.readthedocs.io/en/latest/core_sensors/

So then when you spawn with attachment, location is specified relative to the parent actor.

transform = carla.Transform(carla.Location(x=0.8, z=1.7))
sensor = world.spawn_actor(blueprint, transform, attach_to=my_vehicle)

How to generate data? Scenario runner is an option: https://carla-scenariorunner.readthedocs.io/en/latest/getting_scenariorunner/

Scenario Runner

Traffic manager: https://carla.readthedocs.io/en/latest/adv_traffic_manager/#what-is-the-traffic-manager

The Traffic Manager (TM) is the module that controls vehicles in autopilot mode in a simulation. Its goal is to populate a simulation with realistic urban traffic conditions. Users can customize some behaviors, for example, to set specific learning circumstances.

Other repo, though this uses an older version of CARLA: https://github.com/AlanNaoto/carla-dataset-runner/blob/master/client_bounding_boxes.py

• However, much better learning if you write your things from scratch

Understand how maps work: https://carla.readthedocs.io/en/latest/core_map/#landmarks

get_spawn_points https://carla.readthedocs.io/en/latest/python_api/

List of sensors: https://carla.readthedocs.io/en/latest/ref_sensors/

CARLA AD Challenge

https://leaderboard.carla.org/get_started/

“Sensor data is also available through ROS topics. The topic name is structured as follows: /carla/hero/<sensor-id>. For a complete reference check the CARLA ROS sensor documentation"".

WATonomous

Carla ros bridge stuff: https://git.uwaterloo.ca/WATonomous/wato_monorepo/-/blob/develop/src/simulation/carla_config/scripts/bridge/carla_bridge_manager.py

• It seems that we write some custom things. so that it publishes to /obstacles_3d. this is not out of the box behavior.
• Why cannot we just use this thing? Get all other actors

https://carla.readthedocs.io/en/0.9.13/core_actors/

This is the key function

def get_data(self, filters):
	"""
	Get data specifically from Carla.
	Args:
		filters: list - list of strings that name Carla-specific filters
			e.g. ["filter_facing_ego"] for traffic signs
	Return Type:
		Dict - eg {CARLA_3D_OBSTACLES: obs_list_ros_msg_obj}
	"""
	actors = self.world.get_actors()
	ts_list = TrafficSignListMsg()
	ts_list.header.frame_id = 'odom'
	ts_list.header.stamp = rospy.Time.now()
	traffic_actors = actors.filter("traffic.*")
	if "filter_facing_ego" in filters:
		# only send stop signs that are facing the ego to emulate real life
		applicable_traffic_signs = filter_facing_ego(
			traffic_actors.filter("traffic.stop"),
			self.ego_vehicle.get_transform(), self.world,
			STOP_SIGN_BB_LOC_ADJ,
			STOP_SIGN_BB_EXT_ADJ
		)
	ts_list.traffic_signs = from_carla_traffic_signs(applicable_traffic_signs)
	obs_list = ObstacleListMsg()
	obs_list.header.frame_id = 'odom'
	obs_list.header.stamp = rospy.Time.now()
	obs_list.obstacles = actors_to_obstacles(actors, self.ego_vehicle)
	return {
		CARLA_3D_OBSTACLES: obs_list,
		CARLA_TRAFFIC_SIGNS: ts_list
	}

If you want to do manual control, you should go into the gui_tools docker container, and then launch roslaunch carla_manual_control carla_manual_control.launch. You need to press b to enable manual override, and then use the wasd keys to control the car.

Steps

watod up carla_server
watod up carla_ros_bridge gui_tools # In a separate terminal

Some extra notes

• Rowan was telling me about the odom errors we were running into. You should first start the carla server, which runs on unreal engine. Then, you should launch the carla_ros_bridge, which is essentially the frontend.
• so you can start with
• watod up carla_server (the backend)
• Then, do watod up carla_ros_bridge (the frontend)
• And other instances

The error I was getting gui_tools_1 | [ERROR] [1668642643.334595325, 6.032078996]: Error getting latest time from frame 'base_link' to frame 'odom': Could not find a connection between 'odom' and 'base_link' because they are not part of the same tree.Tf has two or more unconnected trees. (Error code: 2), is because the carla server was not launched yet, the the gui_tools is fetching this transform which is not provided.

The error you were getting from gui_tools was due to a particular transform not being provided.

Installation in Docker

First pull the carla image by running

docker pull carlasim/carla:0.9.13