218 12 The Engineering and Development of Ethics

218 12 The Engineering and Development of Ethics Stage Characteristics Mature Ethics e Formal cognitive basis e Perry’s Relativist and “Constructed Knowledge” stages e The abstraction involved with applying the Golden Rule in practice is more fully understood and manipulated, leading to limited but nonzero deployment of the Cate gorical Imperative e Attention is paid to shaping one’s ethical principles into a coherent logical system e Rationalized, moderated selfishness or selflessness. e Empathy is extended, using reason, to individuals and situations not directly matching one’s own experience e Theory of mind is extended, using reason, to counterin- tuitive or experientially unfamiliar situations e Reason is used to control the impact of empathy on be havior (i.e. rational judgments are made regarding when to listen to empathy and when not to) e Rational experimentation and correction of theoretical models of ethical behavior, and reconciliation with ob- served behavior during interaction with others. e Conflict between pragmatism of social contract orienta- tion and idealism of universal ethical principles. e Understanding of ethical quandaries and nuances develop (pragmatist modality), or are rejected (idealist modality). e Pragmatically critical social citizen. Attempts to main- tain a balanced social outlook. Considers the common good, including oneself as part of the commons, and acts in what seems to be the most beneficial and practical manner. Table 12.5: Integrative Model of the Stages of Ethical Development, Part 2 The main engine of advancement from the conventional to mature stage, we suggest, is robust and subtle integration of the simulative and inferential components. To expand empathy beyond the most obvious cases, analogical inference is needed; and to carry out complex inferences about justice, empathy-guided inference-control is needed. Finally, to advance from the mature to the enlightened stage, what is required is a very advanced capability for unified reflexive inference and simulation. The system must be able to understand itself deeply, via modeling itself both simulatively and inferentially — which will generally be achieved via a combination of being good at modeling, and becoming less convoluted and more coherent, hence making self-modeling easier. Of course, none of this tells you in detail how to create an AGI system with advanced ethical capabilities. What it does tell you, however, is one possible path that may be followed to achieve this end goal. If one creates an integrative AGI system with appropriately interconnected inferential and simulative components, and treats it compassionately and fairly, and provides it extensive, experientially grounded ethical instruction in a rich social environment, then the AGI system should be able to ascend the ethical hierarchy and achieve a high level of ethical sophistication. In fact it should be able to do so more reliably than human beings because of the capability we have to identify its errors via inspecting its internal knowedge-stage, which HOUSE_OVERSIGHT_013134