enough time and unlimited visual aids, a human could express a preference (or

enough time and unlimited visual aids, a human could express a preference (or indifference) when offered a choice between two future lives laid out before him or her in all their aspects. (This idealization ignores the possibility that our minds are composed of subsystems with incompatible preferences; if true, that would limit a machine’s ability to optimally satisfy our preferences, but it doesn’t seem to prevent us from designing machines that avoid catastrophic outcomes.) The formal problem F to be solved by the machine in this case 1s to maximize human future-life preferences subject to its initial uncertainty as to what they are. Furthermore, although the future-life preferences are hidden variables, they’re grounded in a voluminous source of evidence—namely, all of the human choices ever made. This formulation sidesteps Wiener’s problem: The machine may learn more about human preferences as it goes along, of course, but it will never achieve complete certainty. A more precise definition is given by the framework of cooperative inverse- reinforcement learning, or CIRL. A CIRL problem involves two agents, one human and the other a robot. Because there are two agents, the problem is what economists call a game. It is a game of partial information, because while the human knows the reward function, the robot doesn’t—even though the robot’s job is to maximize it. A simple example: Suppose that Harriet, the human, likes to collect paper clips and staples and her reward function depends on how many of each she has. More precisely, if she has p paper clips and s staples, her degree of happiness is Op + (1-0)s, where @ is essentially an exchange rate between paper clips and staples. If @1is 1, she likes only paper clips; if @ is 0, she likes only staples; if 0 is 0.5, she is indifferent between them; and so on. It’s the job of Robby, the robot, to produce the paper clips and staples. The point of the game is that Robby wants to make Harriet happy, but he doesn’t know the value of @, so he isn’t sure how many of each to produce. Here’s how the game works. Let the true value of 0 be 0.49—that is, Harriet has a slight preference for staples over paper clips. And let’s assume that Robby has a uniform prior belief about 6—that is, he believes @ is equally likely to be any value between 0 and 1. Harriet now gets to do a small demonstration, producing either two paper clips or two staples or one of each. After that, the robot can produce either ninety paper clips, or ninety staples, or fifty of each. You might think that Harriet, who prefers staples to paper clips, should produce two staples. But in that case, Robby’s rational response would be to produce ninety staples (with a total value to Harriet of 45.9), which is a less desirable outcome for Harriet than fifty of each (total value 50.0). The optimal solution of this particular game is that Harriet produces one of each, so then Robby makes fifty of each. Thus, the way the game is defined encourages Harriet to “teach” Robby—as long as she knows that Robby is watching carefully. Within the CIRL framework, one can formulate and solve the off-switch problem—that is, the problem of how to prevent a robot from disabling its off-switch. (Turing may rest easier.) A robot that’s uncertain about human preferences actually benefits from being switched off, because it understands that the human will press the off-switch to prevent the robot from doing something counter to those preferences. Thus the robot is incentivized to preserve the off-switch, and this incentive derives directly from its uncertainty about human preferences.’ The off-switch example suggests some templates for controllable-agent T See Hadfield-Menell et al., “The Off-Switch Game,” https://arxiv.org/pdf/1611.08219. pdf. 34 HOUSE_OVERSIGHT_016254