12.9 AGI Ethics As Related to Various Future Scenarios 235

12.9 AGI Ethics As Related to Various Future Scenarios 235 approach does not necessarily need to be augmented with a prior complex of “ascent-safe” moral imperatives at startup time. Developing an AGI with theory of mind and ethical reinforcement learning capabilities as described (admittedly, no small task!) is all that is needed in this case — the rest happens through training and experience as with any other moderate intelligence. 12.9.2 Superintelligent AI: Soft- Takeoff Scenarios Soft takeoff scenarios are similar to capped-intelligence ones in that in both cases an AGI’s progression from standard intelligence happens on a time scale which permits ongoing human interaction during the ascent. However, in this case, as there is no predetermined limit on intelligence, it is necessary to account for the possibility of a superintelligence emerging (though of course this is not guaranteed). The soft takeoff model includes as subsets both controlled- ascent models in which this rate of intelligence gain is achieved deliberately through software constraints and/or meting-out of computational resources to the AGI, and uncontrolled-ascent models in which there is coincidentally no hard takeoff despite no particular safeguards against one. Both have similar properties with regard to ethical considerations: 1. Ethical considerations under this scenario include not only the usual interhuman ethical concerns, but also the issue of how to convince a potential burgeoning superintelligence to: a. Care about humanity in the first place, rather than ignore it b. Benefit humanity, rather than destroy it c. Elevate humanity to a higher level of intelligence, which even if an AGI decided to proceed with requires finding the right balance amongst some enormous considerations: i. Reconcile the aforementioned issues of ethical coherence and group volition, in a manner which allows the most people to benefit (even if they don’t all do so in the same way, based on their own preferences) ii. Solve the problems of biological senescence, or focus on human uploading and the preservation of the maintenance, support, and improvement infrastructure for inor- ganic intelligence, or both iii. Preserve individual identity and continuity of consciousness, or override it in favor of continuity of knowledge and ease of harmonious integration, or both on a case- by-case basis 2. The degree of danger is mitigated by the long timeline of ascent from mundane to super intelligence, and time is not of the essence. 3. Learning that proceeds in a relatively human-like manner is entirely relevant to such human- like intelligences, in their initial configurations. This means more interaction with and imitative-reinforcement-corrective learning guided by humans, which has both positive and negative possibilities. 12.9.3 Superintelligent AI: Hard-Takeoff Scenarios “Hard takeoff” scenarios assume that upon reaching an unknown inflection point (the Singularity point [Vin93, Kur06]) in the intellectual growth of an AGI, an extraordinarily rapid increase HOUSE_OVERSIGHT_013151