did not extinguish innovation as efficiently as it was extinguished among our forebears in

did not extinguish innovation as efficiently as it was extinguished among our forebears in prehistory for thousands of centuries.”® That is why I say that prehistoric people, at least, were barely people. Both before and after becoming perfectly human both physiologically and in their mental potential, they were monstrously inhuman in the actual content of their thoughts. I’m not referring to their crimes or even their cruelty as such: Those are all too human. Nor could mere cruelty have reduced progress that effectively. Things like “the thumbscrew and the stake / For the glory of the Lord” ?° were for reining in the few deviants who had somehow escaped mental standardization, which would normally have taken effect long before they were in danger of inventing heresies. From the earliest days of thinking onward, children must have been cornucopias of creative ideas and paragons of critical thought— otherwise, as I said, they could not have learned language or other complex culture. Yet, as Jacob Bronowski stressed in The Ascent of Man: For most of history, civilisations have crudely ignored that enormous potential... . [C]hildren have been asked simply to conform to the image of the adult... . The girls are little mothers in the making. The boys are little herdsmen. They even carry themselves like their parents. But of course, they weren’t just “asked” to ignore their enormous potential and conform faithfully to the image fixed by tradition: They were somehow trained to be psychologically unable to deviate from it. By now, it is hard for us even to conceive of the kind of relentless, finely tuned oppression required to reliably extinguish, in everyone, the aspiration to progress and replace it with dread and revulsion at any novel behavior. In such a culture, there can have been no morality other than conformity and obedience, no other identity than one’s status in a hierarchy, no mechanisms of cooperation other than punishment and reward. So everyone had the same aspiration in life: to avoid the punishments and get the rewards. In a typical generation, no one invented anything, because no one aspired to anything new, because everyone had already despaired of improvement being possible. Not only was there no technological innovation or theoretical discovery, there were no new worldviews, styles of art, or interests that could have inspired those. By the time individuals grew up, they had in effect been reduced to Als, programmed with the exquisite skills needed to enact that static culture and to inflict on the next generation their inability even to consider doing otherwise. A present-day AI is not a mentally disabled AGI, so it would not be harmed by having its mental processes directed still more narrowly to meeting some predetermined criterion. “Oppressing” Siri with humiliating tasks may be weird, but it is not immoral nor does it harm Siri. On the contrary, all the effort that has ever increased the capabilities of Als has gone into narrowing their range of potential “thoughts.” For example, take chess engines. Their basic task has not changed from the outset: Any chess position has a finite tree of possible continuations; the task is to find one that leads to a predefined goal (a checkmate, or failing that, a draw). But the tree is far too big to 8 Matt Ridley, in The Rational Optimist, rightly stresses the positive effect of population on the rate of progress. But that has never yet been the biggest factor: Consider, say, ancient Athens versus the rest of the world at the time. *° Alfred, Lord Tennyson, Zhe Revenge (1878). 87 HOUSE_OVERSIGHT_016307