Teleoperation: the technology that enables a human to remotely monitor, assist and even drive an autonomous vehicle.

Teleoperation is a seemingly simple capability, yet it involves numerous technologies and systems in order to be implemented safely. In the first article of this series, we established what teleoperation is and why it is critical for the future of autonomous vehicles (AVs). In the second article, we showed the legislative traction and emphasis gained for this technology. In the third and fourth articles, we explained two of the many technical challenges that needed to be overcome in order to enable remote vehicle assistance and operation. The fifth article explained how this is all achieved in the safest possible way.  In this installation, we will get to the most important person in the entire loop, the customer. 

There is an Israeli journalist, Sarah Tuttle-Singer, who writes the stories she hears from taxi drivers. She has so many that she even wrote a book with the best of them. This makes perfect sense. Cabbies spend all day riding around with different passengers and when people are bored, they talk. Invariably, taxi drivers have a million anecdotes to share. Unfortunately with the oncoming transition to autonomy, these stories will go away.

The problem with autonomous vehicles

There is an existential issue with removing a driver from the vehicle. You simply cannot provide 100% service availability nor a satisfactory level of customer experience, not today and not in 50 years. Human interaction is mission-critical. There are many cases where an autonomous system is incapable of responding to the level that the customer desires and is entitled to. The lack of a human behind the wheel not only means that there will be vehicle downtime due to being confused by different situations but that seemingly simple services cannot be delivered. The reason is simple, a machine does not know how to interact with a human the same way that a human does. The customer experience is possibly the biggest issue of them all when it comes to autonomy and they come in four main categories: passenger concerns, emergencies, deliveries, and attended zones.

Passenger concerns

1. Passenger discomfort: Maybe they feel anxious or unsafe. With no human driver to respond to their need the passenger has no one to communicate with; no one to assuage their fears, reassure them, or address the source of the problem.

2. Controlling the A/C or infotainment system: Not every passenger has the same comfort levels or needs, and not everyone knows how to operate certain systems. Issues with temperature control or onboard entertainment are inevitable. Yet there is no one to talk to.

3. Particular drop-off points: Currently, when you arrive near your destination you simply say, “can you drop me off by that door/car/tree/etc?” and typically the driver will do just that. With a self-driving system not only will it not be able to do that but it might bring you to what is technically the right address but not an ideal spot for easy access. This is especially important for older passengers and people going for medical treatments.

4. Passenger infractions: Not all issues stem from the vehicle. What if the passenger is not wearing a seatbelt, or there are too many people crowding the back seat? A human driver would set them straight, and the problem would be solved. Would a robot taxi even be able to recognize these issues let alone handle them?

5. Forgotten items or worse, forgotten baby: In 1999 world-famous cellist Yo-Yo Ma forgot his 2.5M dollar cello in a taxi. Passengers have even forgotten sleeping children. Right now it’s easy to call the driver (whose number is on the receipt). This is no longer an option when there is no driver who can pull over and check the vehicle for people or possessions.

6. Vandalism: Unfortunately not all people are good. In fact, some are rather bad, while others such as teenagers are simply careless and indifferent. Passengers can damage or ruin a vehicle during a ride. Without the watchful eyes of a driver, this behavior would go unchecked. The result is higher maintenance costs, reduced profits, and downtime for the vehicle.

Emergencies

7. Law enforcement: When a vehicle is driving erratically or there is a matter that needs investigation, police use a speaker to alert the driver to pull over. With no driver on board, the police are unable to do their job, and the consequences can be unfortunate.

8. Medical/Ambulances: Similar situation to the police. Perhaps a passenger is having a medical issue and manages to make a 911 call. The vehicle still needs to know where and when to stop to let the medical professionals give the passengers the necessary treatment they need.

Deliveries

9. Wrong/damaged/missing package: When you receive a package from a courier they hand it to you and ask you to sign for it. If the package received is badly damaged, or simply the wrong thing, you can tell them and they handle it. If you are missing an item, they will go back and bring what they forgot. A robot would simply move on to its next destination automatically.

10. Finding the right customer: Today there are only a few vendors with delivery bots. However, these numbers will multiply as the technology improves and production costs drop. Soon there will be many robots with many recipients. A delivery bot might find itself making a drop-off to a location where multiple people are waiting for different orders. To the computer, it is impossible to ascertain which human it needs to make its delivery. This may result in a confused device and a frustrated customer.

Attended zones

11. Content validation: When a truck arrives for a pickup or drop off there is a need for the gate attendant to confirm the records, the contents of the delivery, and to direct the vehicle to its specific parking slot or loading dock. An autonomous system does not respond well to pointing in a direction or verbal commands, nor can it explain why there might be a discrepancy between the information it has and the instructions given by the attendant.

12. Changing orders: Once inside a facility, there may be someone who needs to redirect a vehicle from one task to another. This is especially true with construction zones where one vehicle might have a number of tasks and their order changes. There is no way for that person in the field to communicate these changing needs.

13. Damages: Once the vehicle has arrived there will be some level of vehicle inspection. If there is something wrong with the vehicle or damage has been incurred there is no one to tell.

The solution

For those who read the previous articles in this series, the answer should be obvious. For those who did not, the solution is to have a remote human in the loop. Currently, none of the aforementioned issues is a problem as there are human drivers and couriers. Autonomy cannot resolve these problems. A human being will still be needed to manage these issues in the most effective, efficient, and safe way possible. This is why teleoperation is the only choice. However, as with most solutions, it has its own challenges.

Bringing a human into the loop

When a teleoperation session is triggered it does not simply go to the first available teleoperator (TO). This is for the same reason that when you call customer service there is a routing system that, based on your input, will route you to the agent with the expertise you need. However, unlike a regular call center, the customer does not necessarily have the option to “press three for traffic issues” or “press pound to repeat this message”.

The first possible solution would be for there to be a teleoperation manager (TM) who will answer the calls for assistance first and, upon ascertaining the level of complication and need, route the session to a specific operator. This would be highly inefficient and would mean the TM is not available to do their actual job — managing. Instead, there needs to be an automatic and intelligent way to route the need for human intervention without… human intervention.

When a teleoperation session is triggered the first thing that is automatically ascertained is the source of the request. Did the passenger trigger the need for a TO? Was it a first responder or law enforcement? Or was it the vehicle itself? Each of these situations calls for a different type of response and therefore a different type of TO. Within a given teleoperation team there will be some who are junior and some who are senior, some who are more customer-oriented, and others who are more technical. A senior TO might be authorized for remote assistance and remote driving while their junior counterpart is only allowed to assist. Some TOs might be primarily for customer interaction situations so if the session is triggered by the customer they will be the ones looped in. On the other hand, if the vehicle triggers the session, not because of a confusing traffic situation but because of a technical issue, an entirely different response would be needed.

Remote assistance is complex

We just established that the seemingly simple aspect of who answers which call is complicated. For the challenges in even establishing that connection, refer to our previous articles about network connectivity and video compression.

There is still another challenge. Once the teleoperation session is started, the TO has to understand what exactly is going on. There is a serious amount of information they need to receive that must be layered into their display so they can better understand the situation. This issue multiplies when hopping between numerous vehicles. There have to be processes and tools built-in so that there is minimal delay between the start of a session and the goal of solving the problem for the vehicle and to the customer’s satisfaction.

Enabling autonomy

If autonomous vehicles providers ever want to have mass deployed robot fleets they must ensure the human is in the loop. It is a complicated and complex process from beginning to end and a completely different technology from the autonomy itself. This is why teleoperation vendors exist and why industry leaders like Motional choose to rely on them for this mission-critical function. Anything else and they are not on the right track towards the self-driving future.