7.3 Toward a Formal Characterization of Real-World General Intelligence 137

7.3 Toward a Formal Characterization of Real-World General Intelligence 137 7.3.2 Connecting Legg and Hutter’s Model of Intelligent Agents to the Real World A notable aspect of the Legg and Hutter formalism is the separation of the reward mechanism from the cognitive mechanisms of the agent. While commonplace in the reinforcement learning literature, this seems psychologically unrealistic in the context of biological intelligences and many types of machine intelligences. Not all human intelligent activity is specifically reward- seeking in nature; and even when it is, humans often pursue complexly constructed rewards, that are defined in terms of their own cognitions rather than separately given. Suppose a certain human’s goals are true love, or world peace, and the proving of interesting theorems — then these goals are defined by the human herself, and only she knows if she’s achieved them. An externally- provided reward signal doesn’t capture the nature of this kind of goal-seeking behavior, which characterizes much human goal-seeking activity (and will presumably characterize much of the goal-seeking activity of advanced engineered intelligences also) ... let alone human behavior that is spontaneous and unrelated to explicit goals, yet may still appear commonsensically intelligent. One could seek to bypass this complaint about the reward mechanisms via a sort of “neo- Freudian” argument, via ® associating the reward signal, not with the “external environment” as typically conceived, but rather with a portion of the intelligent agent’s brain that is separate from the cognitive component ® viewing complex goals like true love, world peace and proving interesting theorems as in- direct ways of achieving the agent’s “basic goals”, created within the agent’s memory via subgoaling mechanisms but it seems to us that a general formalization of intelligence should not rely on such strong assumptions about agents’ cognitive architectures. So below, after introducing the pragmatic and efficient pragmatic general intelligence measures, we will propose an alternate interpreta- tion wherein the mechanism of external rewards is viewed as a theoretical test framework for assessing agent intelligence, rather than a hypothesis about intelligent agent architecture. In this alternate interpretation, formal measures like the universal, pragmatic and efficient pragmatic general intelligence are viewed as not directly applicable to real-world intelligences, because they involve the behaviors of agents over a wide variety of goals and environments, whereas in real life the opportunities to observe agents are more limited. However, they are viewed as being indirectly applicable to real-world agents, in the sense that an external intelli- gence can observe an agent’s real-world behavior and then infer its likely intelligence according to these measures. In a sense, this interpretation makes our formalized measures of intelligence the opposite of real-world IQ tests. An IQ test is a quantified, formalized test which is designed to approxi- mately predict the informal, qualitative achievement of humans in real life. On the other hand, the formal definitions of intelligence we present here are quantified, formalized tests that are designed to capture abstract notions of intelligence, but which can be approximately evaluated on a real-world intelligent system by observing what it does in real life. HOUSE_OVERSIGHT_013053