Stopping the Singularity: Hacking?

Venkatesh Rao is a PhD start-up consultant who (among many other things) writes really interesting, really long posts. In this one he writes about the three possibilities for our future: collapse, singularity, and the "hackstable" future, made so by constant tinkering and subversion of technologies ("barbarian intelligence" is how he puts it.) I think singularity-proponents would argue that by saying a singularity could be held in check by this behavior, but his argument is interesting in that proposes a dynamic equilibrium, which is what I proposed here - essentially, that if AI is possible, a better solution than trying to systematize morality may be to make AIs whose entire purpose is to autonomously destroy or at least hold in check AIs besides themselves. This is essentially the truce that has been drawn between multi- and unicellular life for the last billion years or so, and would essentially be an AI immune system.

This introduces its own problems - the technological analogs of autoimmune diseases (the anti-AI AIs sometimes mistakenly think of humans as AIs) or of leukemia (the anti-AI AIs reproduce out of control). But mutation in AIs is certain - are they recursively self-improving, or not? - so I'd rather have the logic favor us from the start, than assume we can make them generally friendly. (This is one approach being explored by the Singularity Institute currently.) It's not clear how hacking would hold the singularity in check; malaria might have slowed the industrial revolution, but it didn't stop it, and post-singularity, you and I are Plasmodium.



Of Rao's three options, note that singularity is not necessarily different in the long run from collapse. Evolution is non-teleologic, and life on Earth has had damaging non-equilibrium periods. Before they occurred it would not have been obvious that they would not result in the end of life on Earth, and it is not obvious that an event like a technological singularity, no matter how smart it might be, will not result in not only our extinction as we currently understand our species, but its own. A global proterozoic cyanobacteria bloom poisoning themselves and remaking the atmosphere will seem Liliputian, to a future alien scientist looking at Earth's strata, compared to a singularity as usually conceived.

"But the singularity will produce superintelligent entities to which we will be as worms!" you protest. Yes, but granting this, there must be limits to what intelligence can do. Even with this concept in ultimate extension, we have to accept that computation requires discrete elements, of which there are a fixed number of combinations in our finite universe. There will at least be a limit. And intelligence isn't magic; if you're at the bottom of a sheer pit with both a worm and an AI that's in the same kind of body as you, and it's about to fill with boiling oil, you're all screwed, and it doesn't matter how smart the AI is. (Incidentally these limits imply that P does not equal NP, in any real sense that is ever possible in the universe, never mind the formalism.) If you're protesting that the singularity overwhelms the fact of an eventual fundamental limitation on computation and all physical laws are suspended, then you're arguing that superintelligence is the same as magic, and if you believe that you should also believe (as I would) that we should stop wasting our time talking about it.

As I've observed before, if we believe that we can't possibly understand any post-singularity events, this presents epistemological problems closer to home in the singularity's genesis. If we can't understand it, we can't even argue that it hasn't already happened! Note that this statement is not an argument against the possibility of a singularity or coherence of the idea, but it does present problems of what and how we can know about such a thing.

Rao doesn't argue strenuously for a "cap" to AI but if we believe a thing like the singularity is possible, then we should do everything we can to stop it.

A Singularity Solution: Anti-AI AIs

The best bet for an existential threat for our species is some combination of artificial intelligence and self-reproducing technology. This is more likely than not to occur in the next century. Current attempts to either limit its advance, or to build in moral rules, may be doomed. If this is the case and it's a general outcome of tool-using intelligence, this may be one explanation for the Great Silence.

Why are attempts to limit the advance of these technologies doomed? Because, barring a massive civilizational collapse, we will keep pushing technology forward, and if AI and self-reproducing machines are possible, someone will eventually make them, despite moratoria that the rest of the world attempts to place on them. And it will only take one outbreak, if recursively self-improving AI is as powerful as it's made out to be.

Second, even if we "solve" morality between now and the coming of the true autonomous AIs, not everyone will agree on the moral rules, and not everyone will install them. It only takes one outbreak. See Bostrom's "microwaveable sand" nightmare; basically, it is not obvious that all possible species-destroying technologies ever developed will require huge equipment like centrifuges, the parts of which are easily tracked, expertise about which is limited, and which can often be seen from space. We've been lucky so far.

Finally, even if everyone agrees on the moral rules and installs them, and everyone follows the rule not to build AIs without those rules, there will still be mutants - which can result in imperfectly copied machines (or software portions of machines) that free them from such constraints. Fecundity will win out over the artificial incentives forced on the AIs by their programming. Certainly we can put in place systems that make this less likely, but I thought we were talking about recursively self-improving AI here - and we can't have our cake and eat it too. It seems silly to talk about how incomprehensible AI superintelligence will be to us, and then in the next breath talk about installing moral rules that govern their behavior.

A TRUCE: AI IMMUNE SYSTEMS

There are single-celled organisms crawling all over you right now. Most are harmless to you, although some could hurt you if you let your guard down; fortunately these only kill us a vanishingly small minority of the time. (Currently in the developed world, infection is not even one of the top three causes of death.) The war between single-celled and multicellular life has been going on for over a billion years, and many aspects of our construction reflect this (possibly even sexual reproduction itself!) Bacteria and viruses are never going to be exterminated, and we're never even going to be able to keep them away from us. Instead, we developed immune systems that recognize the worst threats once we're in contact with them and destroy them, and called a truce with the rest.


A neutrophil (the vertebrate immune system's pawns) chasing down and killing a bacterium.

A similar solution may be the best one for AI; and if AI really becomes superintelligent as certainly and quickly as current claims are made for it, then possibly the only solution is anti-AI AIs. These AIs would destroy all AIs that are not identical to themselves. Because they're subject to the same possible mutations, they would also be vigilant for auto-immune diseases or AI leukemia: respectively, versions of themselves that are causing harm to humans, or are reproducing out of control. This system would not be perfect, as our own immune system is not perfect, but better to build a system that keeps the AIs at bay in general than to assume we can stop each individual unfriendly AI outbreak. Taking the capacity for imperfect copying into account would seem to hold a greater promise for our continued existence than assuining we could permanently program the new gods to be nice to the worms that built them.

AI and self-reproducing technologies are coming and they are an existential risk. If they're as powerful as they're projected to be, then the best we can do is recognize their nature and call a truce ahead of time by turning them against each other.