To Stop AI From Going Rogue, Make It Neurotic

Make AI doubt its own usefulness in order to make sure we can switch it off, according to researchers at the University of California Berkeley.

Mitch Wagner, Executive Editor, Light Reading

June 9, 2017

3 Min Read
To Stop AI From Going Rogue, Make It Neurotic

To reduce risk of AI getting out of control, machines should be made to doubt themselves, according to researchers at the University of California Berkeley.

In other words, make AI neurotic.

In a paper, researchers Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell say that one of the best ways to prevent AIs from misbehaving is to make sure we can turn them off. Even an AI that doesn't care about its own life will want to remain active -- or alive -- to make sure it can continue to be useful. So the solution is to make the AI doubt whether it's useful.

It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off.

It's not enough that the robots will take our jobs. They'll also be neurotic, constantly requiring reassurance.

Figure 1:

Figure 2:

Figure 3:

My colleague Jamie Davis at has some other ideas for personality types that can be programmed into AI to keep them in line:

Conceited Carl: As an AI application, Carl is no danger. Carl spends too long staring into the mirror to fix his hair, down the gym pumping iron or on ASOS trying to figure out what Jon from Love Island was wearing last night. This AI is so concerned about how it appears there is very little risk of world domination. Just tell it that its algorithm looks a little bit flabby and it will be on the virtual cross-trainer in no time.

Also, Hypochondriac Henrietta, Boring Benjamin, and more. Read it here: "How do you stop AI from taking over the world? Make it neurotic of course."

I fear I may be Boring Benjamin. Would you like to discuss the relative merits of Mac to-do list software?

Related posts:

— Mitch Wagner Follow me on Twitter Visit my LinkedIn profile Visit my blog Friend me on Facebook Editor, Enterprise Cloud News

Keep up with the latest enterprise cloud news and insights. Sign up for the weekly Enterprise Cloud News newsletter.

About the Author(s)

Mitch Wagner

Executive Editor, Light Reading

San Diego-based Mitch Wagner is many things. As well as being "our guy" on the West Coast (of the US, not Scotland, or anywhere else with indifferent meteorological conditions), he's a husband (to his wife), dissatisfied Democrat, American (so he could be President some day), nonobservant Jew, and science fiction fan. Not necessarily in that order.

He's also one half of a special duo, along with Minnie, who is the co-habitor of the West Coast Bureau and Light Reading's primary chewer of sticks, though she is not the only one on the team who regularly munches on bark.

Wagner, whose previous positions include Editor-in-Chief at Internet Evolution and Executive Editor at InformationWeek, will be responsible for tracking and reporting on developments in Silicon Valley and other US West Coast hotspots of communications technology innovation.

Beats: Software-defined networking (SDN), network functions virtualization (NFV), IP networking, and colored foods (such as 'green rice').

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like