As artificial-intelligence systems become widespread, the chances grow that their glitches will have dangerous consequences. For example, scientists at the Massachusetts Institute of Technology recently fooled a Google-trained AI program into identifying a plastic toy turtle as a rifle. If a future robot cop or soldier made such an error, the results could be tragic. But researchers are now developing tools to sniff out potential flaws among the billions of virtual “brain cells” that make up such systems.
Many image-recognition programs, car autopilots and other forms of AI use artificial neural networks, in which components dubbed “neurons” are fed data and cooperate to solve a problem—such as spotting obstacles in the road. The network “learns” by repeatedly adjusting the connections between its neurons and trying the problem again. Over time, the system determines which neural connection patterns are best at computing solutions. It then adopts these as defaults, mimicking how the human brain learns.
A key challenge of this technology is that developers often do not know how networks arrive at their decisions. This can make it hard to figure out what went wrong when a mistake is made, says Junfeng Yang, a computer scientist at Columbia University and a co-author of a new study presented last October at a symposium in Shanghai.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Yang and his colleagues created DeepXplore, a program designed to debug AI systems by reverse engineering their learning processes. It tests a neural network with a wide range of confusing real-world inputs and tells the network when its responses are wrong so that it can correct itself. For example, DeepXplore could determine if a camera image fed to a car-driving AI system mistakenly steered a vehicle toward pedestrians. The debugging tool also monitors which neurons in a network are active and tests each one individually. Previous AI-debugging tools could not tell if every neuron had been checked for errors, Yang says.
In tests on 15 state-of-the-art neural networks—including some used in self-driving cars and computer malware detection—DeepXplore discovered thousands of bugs missed by earlier technology. It boosted the AI systems' overall accuracy by 1 to 3 percent on average, bringing some systems up to 99 percent. DeepXplore's technique could help developers build “more accurate and more reliable” neural networks, says Shan Lu, a computer scientist at the University of Chicago, who did not take part in the new research. This approach, Lu adds, “in turn can benefit many scientific research disciplines and our daily lives.”