The algorithms of the Uber driverless car that hit a man to death in March 2018 failed to identify pedestrians crossing the road in the wrong place. Because of this, as well as other features of algorithms that determine the danger of objects in the path of the car, it began to brake only 0.2 seconds before the collision, although he found a person in 5.6 seconds. This follows documents that the U.S. National Transportation Safety Board (NTSB) released before a meeting on November 19, which will determine the cause of the incident.

The accident, which the NTSB is investigating, occurred on March 19, 2018 in Tempe, Arizona. The car with the test engineer at the wheel was moving on the road in the dark at a speed of about 70 kilometres per hour. At this time, at a distance of 110 meters from the nearest pedestrian crossing, 49-year-old Elaine Herzberg was crossing the road in the wrong place, pushing a bicycle loaded next to her, loaded with packets of things. The car hit the woman and after a short time, she died in the hospital from injuries. This was the first death from a car of 3 levels of autonomy.

In May, the NTSB published a preliminary report from which it became known that Herzberg had been detected a few seconds before the collision, but this did not help to avoid the accident due to the erroneous classification of the pedestrian and the disabled emergency braking system. Now the NTSB has published a final set of 40 documents with a total length of over 400 pages, collected during the investigation. It will be used to make a final decision on the causes of the incident at the meeting on November 19. Some of the documents contain new details, in particular, one of them describes in detail the work of algorithms for detecting and classifying objects on the road, as well as predictions of their actions.

As previously known, Uber engineers have reconfigured the Volvo XC90’s regular emergency braking system so that it shuts down completely when driving offline. This was due to a conflict between the radars and algorithms from Volvo and Uber. The company’s developers implemented their own emergency braking system and implemented a second delay, during which the algorithm makes sure of the danger of the situation, and also calculates the detour path, if possible. Engineers added this feature because of the many false positives. If in this second the engineer behind the wheel did not intercept the control, and the algorithm did not stop identifying the object on the way as dangerous, the system begins braking itself.

Another feature of the implementation of the algorithms that influenced the outcome of what happened is how exactly the classification of objects on the path and the forecast of their movement took place. Classification algorithms were trained to recognize pedestrians only near pedestrian crossings. In other cases, it attributed the object to vehicles or unidentified objects.

In the case of vehicles, algorithms predicted their further movement in accordance with standard scenarios. For example, if an algorithm recognized an object as a car, it initially assumed that it would go forward in its lane, and then analyzed the history of data from sensors and predicted the trajectory. At the same time, the motion forecast was calculated only from the moment when the algorithm assigned a certain class to the object.

In many ways, this led to the fact that the car, which saw a person 5.6 seconds before the collision, could not reduce its speed in advance. The fact is that due to the fact that the algorithm did not know how to recognize pedestrians, it changed the class of the object many times as it approached it and thereby reset the history of readings. Moreover, at the moments when it defined Herzberg as an unidentified object, it mistook it for a static object and did not calculate the future trajectory at all.

Because of this, the algorithm finally assigned Herzberg to the “bicycle” class and calculated that it would be in the way of movement, only 1.2 seconds before the collision. At this point, the system started the braking procedure, but in fact it was delayed for a second due to the settings. In 0.2 seconds, the car began to brake and notified the driver, but it looked at the smartphone and pressed the brake pedal 0.7 seconds after the impact.

The NTSB documents contain many other details, and it is also noted that since the accident, Uber has redesigned the technical aspects of the operation of drones, in particular, the emergency braking system. Calculations showed that with the new version of the software, braking would have started more than four seconds earlier. In addition, the agency noted that Uber has generally changed the security system in its project for the development and testing of unmanned vehicles. In December 2018, companies were allowed to continue testing unmanned vehicles on the road.

The Uber driverless car accident has affected the trials of many developers. For example, Uber itself, as well as Toyota and NVIDIA, stopped testing shortly after the incident. In addition, the incident has generally affected the pace of drone development and testing programs for many companies in the industry.