Gordon Briggs, a postdoc at the U.S. Naval Research Laboratory, talks about the article he and Matthias Scheutz, director of the Human Robot Interaction Laboratory at Tufts University, wrote in the January Scientific American titled "The Case for Robot Disobedience."

Steve Mirsky:    Welcome to Scientific American’s Science Talk, posted on December 21, 2016. I’m Steve Mirsky. On this episode -

Gordon Briggs:  The robotic agents in that series – or at least they alluded to specific agents having very specific limitations into what they're allowed or not allowed to do. They implied, for instance, that the agents were – they had hard limitations about using lethal force, for instance.

Mirsky:               That’s Gordon Briggs talking about West World. He's a post doc as the U.S. Naval Research Laboratory and he's the co-author of an article in the January issue of Scientific American titled, "The Case for Robot Dissentience." He recently earned a joint doctorate in computer and cognitive science from Tufts University working with his co-author, Matthias Scheutz, director of the Human Robot Interaction Lab at Tufts. The subhead for their article reads, "Don't worry about defiant machines. Devious masters and misunderstood commands are a bigger threat." I spoke to Briggs by phone.

                           So it's sometime in the future and I'm in my house with my friend and my robot. My friend is a funny guy and my robot is sitting there. My robot is programmed to protect me, of course, and my friend is cracking jokes and I just say instinctively, "This guy is killing me," and the robot attacks my friend and subdues him because it was under the impression, based on what I said, that I was in great danger. So it seems to me that what you're really trying to do is prevent that kind of thing from happening.

Briggs:               That's certainly one of the many things that we don't want to happen. That's a funny case in that the robot has to make an inference based off of the sort of side comment, so that requires a bit of sophisticated reasoning in and of itself before it could even derive a goal to protect you from the facetious threat. In general, we're interested in cases where the robot should actually not act as opposed to act, and a lot of research is directed in giving robots new capabilities and commanding them in natural language to access those capabilities, but there's this reverse problem of just because the robot can do something even as complex as intervening in a social situation, it isn't necessarily always appropriate to do so. We're interested in developing mechanisms to allow the robot to actually reason about when it is and is not appropriate to act and what the appropriate response should be.

Mirsky:               Let me read a short paragraph from the story. "For the time being, super intelligent machines that pose an existential thread to humanity are the least of our worries. The more immediate concern is how to prevent robots or machines with rudimentary language and AI capabilities from inadvertently harming people, property, the environment, or themselves." We can get back to this later, but you open that paragraph with, "For the time being," so maybe we'll talk about what may be on the horizon later that you don't talk about in the article.

Briggs:               Sure.

Mirsky:               We have a real issue here that AI is starting to become more and more a part of our lives. Just the idea of our driverless cars, that seems to be something that is really on the horizon. We need to really address this issue of giving AI the power, to some degree, interpret things and come up with a response that makes sense in a context even if it's not directly the literal thing that it might have heard to do.

Briggs:               Right, and the driverless cars is one good example. I think in the piece, we talk about an instance where perhaps the person is running late and sort of insists that the car, for instance, maybe drive faster when that may not be the safest thing to do. In some sense, there needs to be some sort of reasoning about when it is or is not appropriate to heed the literal commands of the human that's in charge of the machine and what the sort of deeper intentions are. Clearly, they don't intend to get in an accident. The human may not be fully aware of all the details of the scenario to realize what the potential hazards are, or they may just be more willing to take risks than the system or the system designers might be comfortable with. There's a variety of different considerations that have to go into whether or not an autonomous system actually accepts a human directive beyond whether or not it is just capable or knowledgeable of how to actually execute these directives.

Mirsky:               Do we have a template for this kind of a situation with the military? Military people are trained to follow orders, but you're also supposed to exercise judgement when you are given what you perceive to be an illegal order.

Briggs:               Right. Actually, a lot of this interest in robot ethics and machine ethics, I think came to sort of the public forefront when – there was a researcher down at Georgia Tech, Ronald Arken, who really did a lot of the initial work. His research was on enabling autonomous systems to be able to appropriately decide when or when not to use lethal force in a military context. He proposed something called the Ethical Governor, which was something that would take in the best knowledge that the robot has of the situation and see how this particular comports with its sort of formalized representation of the laws of war and the rules of engagement. It would decide whether or not to carry out that order.

Mirsky:               Yeah, and I take it that the term Governor in this case comes from the automobile use of the word Governor or in mechanical devices in general rather than –

Briggs:               Yeah. He made the analogy to the governor on steam engines.

Mirsky:               Right, to keep it from going too fast. That's what I've been thinking about as I read the whole article was about you're really just trying to install governors all over the place to keep things from getting out of control in an AI way.

Briggs:               Right.

Mirsky:               There's an experiment that you conducted that you discuss in the article that's pretty interesting where you have a robot who builds a tower and then you have people instruct the robot to knock the tower down that the robot just built. In one case, the robot just knocks it down, but in another case, the robot is programmed to plead with the people telling it to knock the tower down. The robot pleads with them, "Please, I worked really hard on it. Please don't make me knock it down," etc.

Ultimately, the robot will follow the command if the people are persistent enough in their order giving. But what's really interesting is that about half the test subjects ultimately decided to let the robot leave the tower standing. What's really interesting is this little end of the paragraph, and you don't go into it in the article – the reasoning here, if you know what it is, I'm just curious because there's not space in the article for it. But you wrote here, "Most of the test subjects in the group reported some level of discomfort when they ordered the robot to knock down the red tower." This is after the robot is pleading with them, "Please don't make me knock it down." Here's the really interesting part. "We were surprised to find, however, that their level of discomfort had little bearing on their decision to leave the tower standing or not." So the people who felt uncomfortable were apparently just as likely to continue to tell the robot to knock it down than were the people who didn't have any feelings. So what does that tell us?

Briggs:               That's right. I think it tells us that this particular scenario was a bit more complex than we had first thought. Our initial hypothesis related to the discomfort level – the level of agency that the person would describe to the robot. Whether they thought it was – how machine-like or how human-like or animal-like they thought the robot was would have a strong effect on whether or not they decided to push the robot despite the sort of protest and sort of affective or emotional display that the robot produced. And we found that that really actually didn't strongly affect people's behaviors. We actually did a variety of follow-ups and we're still actually running some follow-ups to tease out what really does cause people to either refrain from forcing the robot to do this or they just give up and kind of heed the robot's protests. One of the directions that it's starting to point to is that it's really just about what people infer about the capabilities or the social context of the situation. If they believe that the robot is just – because it protests multiple times – incapable of carrying out the task – if it's been pre-programmed to reject it, then they may not push it enough times to actually override that temporary…

                           On the flip side – that's something something they might infer if they believe that the robot has low agency, that it's just a machine that has been programmed to do something. Now some subjects ascribe high agency to it because it can communicate in natural language and it has relatively human-like sort of effective capabilities within the confines of this very simple scenario and script. Some people infer high levels of agency, but in that case, what might be pushing people to force the robot is sort of a sense of social obligation like, "Given this situation, you have been tasked to be my subordinate. I don't care if you don't like it. You're going to knock down that tower." So there's multiple different explanations as to why people are either refraining or not refraining based off of the level of agency they ascribe to the robot. I think it just shows that a very simple interaction with this is actually relatively complicated and you have to kind of tease out precisely which of these multiple explanations really is applicable for any given subject.

Mirsky:               Right. What I took to be a deep empathy on the part of some people might be just their reaction to, "Well, no matter how many times I tell it to do it, it's not going to do it, so I'm not going to waste my time."

Briggs:               Right. Exactly.

Mirsky:               Even if they're feeling something about it because of the humanoid-like appearance and behavior of the robot, they're still going to fall back on the, "Well, I keep clicking the button and no pellet is coming out, so I'm going to give up now."

Briggs:               Exactly. Beyond that, it's also this disconnect between we in have instinctually have an emotional response to emotional displays from inanimate objects, but that doesn't necessarily mean we act upon that. We can understand that this – sometimes people can understand, "Well, I'm just getting this reaction because it's sort of instinct," but that doesn't necessarily affect their decision-making in a particular interaction context.

Mirsky:               Sure. In the article, you cite the movies 2001 and of course the famous Hal, "I'm sorry, Dave. I'm afraid I can't do that," and the movie Ex Machina, which I saw, which was thought provoking. I assume that West World aired too late for you to do any commentary on the piece.

Briggs:               Sadly, yes, otherwise that would have been a good series to reference.

Mirsky:               Do you have any thoughts about West World?

Briggs:               There's a lot of different things to talk about there. It's hard to really articulate on the fly. I thought it was interesting, though, that they did have the robotic agents in that series – they did have very explicit – or at least they alluded to specific agents having very specific limitations into what they're allowed or not allowed to do. They implied, for instance, that the agents had hard limitations about using lethal force, for instance, where it seemed like there were some agents that even if they wanted to use lethal force were completely incapable of doing that. Which is very similar to sort of considerations that – sort of explicit prohibitions that people are thinking about in terms of the robotic domain in real life.

Mirsky:               Let's go back to this clause in one of the sentences where you said, "For the time being, super intelligent machines that pose an existential threat to humanity are the least of our worries." So, "For the time being," that just sits there like an ominous cloud.

Briggs:               I guess I use that phrase not to be ominous, but rather to not – I don't want to eliminate the possibility of such a thing ever happening, so I guess I was using that phrase as a way to avoid coming down hard on one side or the other about whether that concern is inherently silly or eminently reasonable. I did want to highlight the fact that if it is possible at all, then that's something that's actually a more long-term future concern. I don't believe that in the near term we should be focused on that because I think there's a lot of near term challenges with regards to robot ethics that do deserve our attention and attention to something of a limited resource. You shouldn't completely discount it, but there's a lot of near term issues that I think we should focus on.

Mirsky:               Very good. It's interesting because my interpretation of, "For the time being," versus your original meaning is a good example of just how difficult it is to make intention crystal clear even when two humans are talking to each other.

Briggs:               That's absolutely correct. Actually, a lot of what I'm interested in my research is conversational pragmatics. That's an entire field about how do people actually infer the appropriate intentions from speech when two strings of words can have vastly different meanings depending on conversational context, and who is speaking, and common ground, and many other things.

Mirsky:               Absolutely. This was great. Thanks for talking to us. I hope people will read the article in the January issue of Scientific American, "The Case for Robot Disobedience." Gordon Briggs, thanks very much.   

Briggs:               Thank you.

Mirsky:               That’s it for this episode. Get your science news at our Web site, www.ScientificAmerican.com, where you can read Pauline Gagnon's article, "The Forgotten Life of Einstein's First Wife, Physicist Mileva Einstein," and check out the latest episode of the nature podcast where a surprise guest joins the nature team to play a game of science taboo. Surprise, I'm the guest. And follow us on Twitter where you’ll get a tweet whenever a new item hits the Web site. Our Twitter name is @sciam.  For Scientific American Science Talk, I’m Steve Mirsky. Thanks for clicking on us.