This post was written by Amit Rosenzweig, CEO of Ottopia.

Teleoperation: the technology that enables a human to remotely monitor, assist and even drive an autonomous vehicle.

Teleoperation is a seemingly simple capability, yet it involves numerous technologies and systems in order to be implemented safely. In the first article of this series, we established what teleoperation is and why it is critical for the future of autonomous vehicles (AVs). In the second article, we showed the legislative traction and emphasis gained for this technology. In the third and fourth articles, we explained two of the many technical challenges that needed to be overcome in order to enable remote vehicle assistance and operation. In this article, we will explore how this is all achieved in the safest possible way. 

More than a decade ago, the major AV companies made a promise. They claimed that autonomous vehicles would by now be completely self-sufficient. Human driving was obsolete. As the years pass, we continue to see how this goal is elusive, and that there will always be the need for a human to be kept in the loop. The initial response to this was remote driving.

Remote Driving? Major danger

Teleoperation was originally a system that overrides the autonomy of a vehicle and allows a human to manually drive it remotely. Essentially it would replace all self-driving functions and safety systems with a remote driver. This would appear to make a degree of sense. Currently, the solution for unknown situations, aka edge cases, is to put a “safety driver” in the driver’s seat. This way, when the autonomy does not know what to do and gets stuck, the human can manually solve the problem by driving the car for just a few seconds. By enabling the human driver to be in a remote location, they can monitor and solve problems for multiple vehicles, thereby cutting down on driver costs.

Chances are when people first envisioned this remote driving, they assumed we would have perfect and fully immersive virtual reality with zero latency as seen in a sci-fi movie like Black Panther. Unfortunately, there are critical shortcomings with regard to remote driving. As it is, from the instant a driver recognizes an obstacle in the road until their foot hits the brake pedal – brake reaction time – it takes about 0.7 seconds. This means that at a speed of only 30 mph, which translates to 44 feet per second, over 30 feet of braking distance are needed to prevent a collision. This is if the driver is IN the vehicle, traveling at ONLY 30 mph, and the car stops on the spot.

For a remote driver, one must factor in at least a few fractions of a second in latency plus the lack of haptic feedback. In other words, the brake reaction time alone is at least 0.8 seconds, with a minimum of 35 feet needed to avoid a collision at 30 mph. And this does not even factor in braking distance. Maybe this is why in a different sci-fi movie, Guardians of the Galaxy 2, one can see how remote pilots are inferior to those onboard the ship.

Clearly, humans cannot be allowed to drive a vehicle from a remote location. At least not on their own.

Advanced Teleoperator Assistance System (ATAS ®): the first transformation for teleoperation

Yes, originally the teleoperation system would shut off the autonomy stack and enable a person to drive the vehicle, but why? Why would you shut off this incredible piece of technology that already knows how to sense, react and respond in ways a person will never be able to do? This is why the second stage of teleoperation involved systems like ATAS® (an Ottopia registered trademark).

Like the more familiar ADAS (Advanced Driver Assistance System) the purpose of ATAS® is to work with the (remote) driver while leveraging the existing safety functions enabled by the vehicle’s autonomous capabilities. The main directive of an ATAS® is to prevent collisions. There are two main ways to do this, both made possible by the autonomy stack.

The first is collision warning. At every given moment, the powerful LiDAR, perception, and computation capabilities are ascertaining each and every object in the field of view of the AV. As the vehicle progresses on its way, the system identifies the speed and trajectory of the vehicle in addition to things that may pose a safety hazard. The teleoperator display has a layer that shows their heading and can alert if anything might be a reason to slow down, stop or circumnavigate the particular obstacle. This system helps compensate for the reactive shortcomings of a human driver while still allowing them to make the important decisions of how to get where they need to go.

The second is collision avoidance. The ultimate safety decision-making power does not and cannot lie with the human driver. Yes, the human is subject to what the autonomy decides is safest! This may seem backwards until you remember that the vehicle is in the moment. It has instant perception abilities. It sees the oncoming crash before any human ever could. Furthermore, even if the human driver could see the potential risk, it is possible they are distracted or blinded or otherwise incapable of recognizing the impending danger. That is why, only with regard to braking in safety situations, the vehicle and its corresponding autonomy system must make the decision to stop the vehicle and prevent a disaster.

Clearly, a remote driver must have a system like ATAS® in order to ensure the safety of those in an AV and those around it. However, there remains serious room for improvement.

Tele-assistance. The final form?

Tele-assistance, also known as remote vehicle assistance (RVA), high-level commands, or indirect control – is when the operator gives certain orders to the AV without directly deciding how it completes that task. Tele-assistance helps reduce many of the risks involved in remote driving, even with ATAS®. Tele-assistance is also dramatically more efficient in terms of how many operators are needed.

This is how Tele-assistance works: In the traditional teleoperation situation, an AV would be driving along when it encounters an event which it does not know how to handle. It pulls over to the safest possible spot, stops, and triggers an alert for human intervention. That human would link in, observe the situation, and decide on how best to remedy the problem. Instead of putting their hands on a steering wheel and feet on pedals, the operator will choose from a menu of commands they can give to the vehicle to guide it out of its predicament.

Examples of such commands include path choosing – where the operator selects one of a few offered choices for an optimal path forward; path drawing – where the operator makes a custom path for the AV to follow; and object override – recognizing when the seeming obstacle is not a problem (e.g., a small cardboard box in the middle of the lane) and, in fact, the vehicle can simply continue on its way.

Traditional teleoperation created more problems than it solved. It is hubristic to claim that a human can remote-drive any normal-sized automobile or truck without any assistance or dedicated safety technology. While humans are required to handle situations confronted by autonomy, the solution for driving is ideally assistance, and at the very least, driving with a safety system like ATAS®.

When tele-assistance is coupled with maximized network connectivity and dynamic video compression, as described in the previous two articles, autonomous vehicles can be commercially deployed in the safest and most efficient manner.

Amit Rosenzweig is the CEO & Founder of Ottopia