the paradox of automation: on automation and self-driving cars

“Crash: how computers are setting us up for disaster,” an article by Tim Harford in the Guardian, is about how automation diminishes our skills. The article is an excerpt from Harford’s new book, Messy: The Power of Disorder to Transform Our Lives.  You can listen to an audio version of the article here.

I knew that this issue of technology eroding skills exists and is not new. Before there were writing systems, history was oral and information was passed down verbally. As a result, being able to memorize large amounts of information was a profoundly useful skill. Once we could could write information down and look it up later, the skill of memorization became less useful. Technology makes certain skills obsolete. That frees us up to develop new and more complex skills, but paradoxically, this makes us vulnerable.

From the article, I learned that these issues lead to the paradox of automation:

It applies in a wide variety of contexts, from the operators of nuclear power stations to the crew of cruise ships, from the simple fact that we can no longer remember phone numbers because we have them all stored in our mobile phones, to the way we now struggle with mental arithmetic because we are surrounded by electronic calculators. The better the automatic systems, the more out-of-practice human operators will be, and the more extreme the situations they will have to face. The psychologist James Reason, author of Human Error, wrote: “Manual control is a highly skilled activity, and skills need to be practised continuously in order to maintain them. Yet an automatic control system that fails only rarely denies operators the opportunity for practising these basic control skills … when manual takeover is necessary something has usually gone wrong; this means that operators need to be more rather than less skilled in order to cope with these atypical conditions.”

I have seen the paradox of automation in my research. One of my areas of research is in emergency medical services. Here, paramedics and emergency medical technicians implement a variety of medical techniques and procedures. One way to improve system performance — which reflects response times — is to increase the number of service providers. More service providers means a greater likelihood of having someone readily available and nearby to the next call, which in turn increases the likelihood of short response times. This is normally a good thing. The downside is that each service provider treats fewer patients and their skills erode because they rarely have to implement some of the procedures, so when they do, they do not do it effectively. The medical literature confirms this (it’s true that if you don’t use it, you lose it). This issue is one reason why medical personnel undergo regular training, but ensuring regular practice in the field seems to be the best way to go.

The paradox of automation will be an issue with self-driving cars.

The US Department of Transportation  adopted SAE International’s six levels of automation for autonomous cars, which provides a useful framework for discussing the future of self-driving cars:

• Level 0 – No Automation: The full-time performance by the human driver of all aspects of the dynamic driving task, even when enhanced by warning or intervention systems
• Level 1 – Driver Assistance: The driving mode-specific execution by a driver assistance system of either steering or acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task
• Level 2 – Partial Automation: The driving mode-specific execution by one or more driver assistance systems of both steering and acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task
• Level 3 – Conditional Automation: The driving mode-specific performance by an Automated Driving System of all aspects of the dynamic driving task with the expectation that the human driver will respond appropriately to a request to intervene
• Level 4 – High Automation: The driving mode-specific performance by an Automated Driving System of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene
• Level 5 – Full Automation: The full-time performance by an Automated Driving System of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver

Levels 0-2 require a driver performing major driving functions. Drivers are necessary until Level 5 is reached. In the future, cars will be partially or mostly automated and will require someone to drive and/or regularly intervene, and therefore, drivers will have to learn how to manage a partially-autonomous car instead of driving it.

The idea of automation and design is discussed in the New York Times Magazine article “Rev-up: imagining a 20% driving world.” John Lee, a professor in my department at UW-Madison, discusses the paradox of automation when it comes to driving a car that is partially autonomous. “Driving and managing the automation that is helping you drive are two quite different skill sets. Automation-management skills need to be learned as much as driving skills,” he says.

The paradox of automation is discussed in the 99% Invisible podcast.

One of my favorite articles from The Onion