The Engineering and Development of Ethics

Chapter 12 The Engineering and Development of Ethics Co-authored with Stephan Vladimir Bugaj and Joel Pitt 12.1 Introduction Most commonly, if a work on advanced AI mentions ethics at all, it occurs in a final summary chapter, discussing in broad terms some of the possible implications of the technical ideas pre- sented beforehand. It’s no coincidence that the order is reversed here: in the case of CogPrime, AGLethics considerations played a major role in the design process ... and thus the chapter on ethics occurs near the beginning rather than the end. In the CogPrime approach, ethics is not a particularly distinct topic, being richly interwoven with cognition and education and other aspects of the AGI project. The ethics of advanced AGI is a complex issue with multiple aspects. Among the many issues there are: 1. Risks posed by the possibility of human beings using AGI systems for evil ends 2. Risks posed by AGI systems created without well-defined ethical systems 3. Risks posed by AGI systems with initially well-defined and sensible ethical systems eventu- ally going rogue — an especially big risk if these systems are more generally intelligent than humans, and possess the capability to modify their own source code 4, the ethics of experimenting on AGI systems when one doesn’t understand the nature of their experience 5. AGTrights: in what circumstances does using an AGI as a tool or servant constitute “slavery” In this chapter we will focus mainly (though not exclusively) on the question of how to create an AGI with a rational and beneficial ethical system. After a somewhat wide-ranging discussion, we will conclude with eight general points that we believe should be followed in working toward "Friendly AGI" — most of which have to do, not with the internal design of the AGI, but with the way the AGI is taught and interfaced with the real world. While most of the particulars discussed in this book have nothing to do with ethics, it’s important for the reader to understand that AGI-ethics considerations have played a major role in many of our design decisions, underlying much of the technical contents of the book. As the materials in this chapter should make clear, ethicalness is probably not something that one can meaningfully tack onto an AGI system at the end, after developing the rest — it is likely infeasible to architect an intelligent agent and then add on an “ethics module.” Rather, ethics is something that has to do with all the different memory systems and cognitive processes that 205 HOUSE_OVERSIGHT_013121