might alter its motion when carrying something heavy, to emphasize the difficulty it has

might alter its motion when carrying something heavy, to emphasize the difficulty it has in maneuvering heavy objects. The more that people know about the robot, the easier it is to coordinate with it. Achieving action compatibility will require robots to anticipate human actions, account for how those actions will influence their own, and enable people to anticipate robot actions. Research has ,ade a degree of progress in meeting these challenges, but we still have a long way to go. The Value Alignment Problem: People hold the key to the robot’s reward function. Progress on enabling robots to optimize reward puts more burden on us, the designers, to give them the right reward to optimize in the first place. The original thought was that for any task we wanted the robot to do, we could write down a reward function that incentivizes the right behavior. Unfortunately, what often happens is that we specify some reward function and the behavior that emerges out of optimizing it isn’t what we want. Intuitive reward functions, when combined with unusual instances of a task, can lead to unintuitive behavior. You reward an agent in a racing game with a score in the game, and in some cases it finds a loophole that it exploits to gain infinitely many points without actually winning the race. Stuart Russell and Peter Norvig give a beautiful example in their book Artificial Intelligence: A Modern Approach: rewarding a vacuuming robot for how much dust it sucks in results in the robot deciding to dump out dust so that it can suck it in again and get more reward. In general, humans have had a notoriously difficult time specifying exactly what they want, as exemplified by all those genie legends. An AI paradigm in which robots get some externally specified reward fails when that reward is not perfectly well thought out. It may incentivize the robot to behave in the wrong way and even resist our attempts to correct its behavior, as that would lead to a lower specified reward. A seemingly better paradigm might be for robots to optimize for what we internally want, even if we have trouble explicating it. They would use what we say and do as evidence about what we want, rather than interpreting it literally and taking it as a given. When we write down a reward function, the robot should understand that we might be wrong: that we might not have considered all facets of the task; that there’s no guarantee that said reward function will a/ways lead to the behavior we want. The robot should integrate what we wrote down into its understanding of what we want, but it should also have a back-and-forth with us to elicit clarifying information. It should seek our guidance, because that’s the only way to optimize the true desired reward function. Even if we give robots the ability to learn what we want, an important question remains that AI alone won’t be able to answer. We can make robots try to align with a person’s internal values, but there’s more than one person involved here. The robot has an end-user (or perhaps a few, like a personal robot caring for a family, a car driving a few passengers to different destinations, or an office assistant for an entire team); it has a designer (or perhaps a few); and it interacts with society—the autonomous car shares the road with pedestrians, human-driven vehicles, and other autonomous cars. How to combine these people’s values when they might be in conflict is an important problem we need to solve. AI research can give us the tools to combine values in any way we decide but can’t make the necessary decision for us. 101 HOUSE_OVERSIGHT_016904