Robots can already discern and react to speech thanks to voice-recognition software such as the iPhone's Siri. But “smart” machines still struggle with most other sounds. “In some sense, it's almost a simpler problem, but there hasn't been a lot of work on noise in the environment,” says roboticist Joseph Romano of Rethink Robotics in Boston. “It hasn't been in the loop for robotic feedback.”
Now Romano is letting robots listen in on more than our conversations. He and his collaborators at the University of Pennsylvania have created a software tool called ROAR (short for robotic operating system open-source audio recognizer) that allows roboticists to train machines to respond to a much wider range of sounds. As described in a recent issue of Autonomous Robots, the tool's chief requirement is a microphone.
To begin training, the robot's microphone first captures ambient sounds, which ROAR scrubs of noisy static. Next the operator teaches ROAR to recognize key sounds by repeatedly performing a specific action—such as shutting a door or setting off a smartphone alarm—and tagging the unique audio signature while the robot listens. Finally, the program creates a general model of the sound of each action from that set of training clips.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
The group tested ROAR on a one-armed robot, improving the machine's ability to complete specific tasks. In one scenario, the robot attempted to autonomously grasp and activate an electric drill. Without any sonic feedback, the robot only succeeded in nine out of 20 attempts, but its success rate doubled while using ROAR. If after grasping, the robot did not hear the whir of the electric motor, it adjusted its grip and tried again.
The next step is to ensure the system works in loud environments. Integrating audio into a robot's feedback loop alongside visual and tactile cues could someday allow robotic nurses to rapidly respond to cries for help or enable factory robots to react when something breaks. Although the technology is in early stages, Romano thinks the potential is enormous. “We haven't even begun to explore what we can do,” he says.