Active Learning, Data Selection, Data Auto-Labeling, and Simulation in Autonomous Driving — Part 7

The last company, Aurora!

Aurora

Aurora, on the other hand, takes a slightly different approach. Rather than blindly pushing for increased mileage, they’ve maintained a focus on collecting high-quality real-world data and extracting the maximum value from each data point. For instance, they amplify the impact of real-world experience by identifying interesting or novel events and incorporating them into their Virtual Testing Suite, where they are used to continuously improve the Aurora driver.

source

However, not all real-world events are amenable to simulation in virtual environments. For instance, the exhaust of a vehicle may be of interest to an object detection system. Thus, using real-world data from such scenes to train the perception system to recognize and ignore exhaust can result in a more enjoyable driving experience.

source

The on-road events that they turn into virtual tests come from two sources:

• Copilot annotations: Vehicle operator copilots, who provide support from the passenger’s seat, routinely flag experiences that are interesting, uncommon, or novel. They frequently recreate these in their Virtual Testing Suite to familiarize the Aurora Driver with a variety of road conditions.
• Disengagements: when their vehicle operators proactively retook control when they suspected an unsafe situation was about to occur or when they disliked the way the vehicle was driving.

Their Virtual Testing Suite is a complementary collection of tests that evaluate the software’s functionality at every level. As a result, they transform real-world events into one or more of the following virtual tests (read more here):

• Perception tests: Consider the following scenario: a bicyclist passes one of their vehicles. Specialists review the event’s log footage and then label items such as the object category (bicyclist), the velocity (3 mph), and so on. They can then use this “absolute truth” to assess the ability of new versions of perception software to accurately determine what occurred on the road. Here is an example of the labeling procedure:

source

• Manual driving evaluations: They compare the planned trajectory of the Aurora Driver to the actual trajectory of the vehicle operator and test whether their motion planning software can accurately forecast what a trained, expert driver would do in complex situations: (comparing a vehicle operator’s trajectory (blue) to the intended trajectory of the Aurora Driver (green) during a right turn).

source

• Simulations: Simulations are virtual representations of the real world in which the Aurora Driver can be tested in numerous permutations of the same situation. Additionally, simulations enable them to simulate a wide variety of interactions between the Aurora Driver and other virtual world actors. For instance, how will a jaywalker react when the Aurora Driver comes to a halt? And then, how do the simulation’s other actors react when the jaywalker crosses the street?

source

By utilizing their online-to-offline pipeline, they convert on-road events into virtual tests:

source

Let’s review this process by an example of a disengagement that helped the Aurora Driver learn how to nudge (when the Aurora Driver adjusts its trajectory to move around obstacles).

• On-Road Event: Vehicle operators annotate disengagements and highlight scenes that are unusual, novel, or interesting. The Aurora Driver hesitates in this situation to nudge around a vehicle that abruptly veers off the roadway and into an on-street parking space. To avoid causing traffic disruptions, the trained vehicle operators quickly take control and drive around the parked vehicle.

• Triage: The triage team conducts an examination of on-road incidents and provides an initial diagnosis. For instance, AV must nudge a vehicle to pull over and come to a complete stop. #motionplanning
• Create virtual tests: They develop one or more simulated tests, which may include perception tests, manual driving evaluations, and/or simulations. They used it as the basis for 50 new nudging simulations, which included a recreation of the exact scene from the disengagement log footage and variations created by altering variables such as the speed of the vehicle in front of the ego car.

source

• Iterate: The diverse Virtual Testing Suite enables engineers to fine-tune new and existing capabilities. The engineering team fine-tuned the Aurora Driver’s ability to nudge using the nudging simulations inspired by this disengagement, as well as numerous other codebase tests (unit and regression), perception tests, and manual driving evaluations.
• Test on the road: They put enhancements to the test in the real world and continue to collect useful data. Here is the Aurora Driver nudging gracefully through a complex situation:

Conclusion

We investigated several active learning techniques and the approaches taken by various companies to collect, select, label, and simulate useful and informative data and scenarios to train their algorithms to handle a variety of real-world situations.

Thank you for taking the time to read my post. If you found it helpful or enjoyable, please consider giving it a like and sharing it with your friends. Your support means the world to me and helps me to continue creating valuable content for you.