Roko's Basilisk in the Form of a Metal Song

When Metallica writes a song about the singularity, it's probably close. "Stop breathing, and dedicate to me - stop dreaming, and terminate for me."

New Approaches on What the Fermi Paradox Means for the Future of Humanity

I was lucky to attend a video lecture by James Miller, economist at Smith College, facilitated by Joshua Fox. Thanks for having this event! I contacted James to let him know I would be posting this and to let him proofread my recapitulation of his argument so as to avoid mis-paraphrasing him; my thanks to him for taking the time to correct me on several points. Of course any errors are mine.

Much of this is familiar terrain for those of us who spend our time considering X-risk and the Fermi paradox. Miller's thesis is that we are at a critically important point in human history, a window where we think that in the near future we can start colonizing the galaxy (the year 2614 at earliest, by this calculation) but at the same time where we are smart enough to destroy ourselves. Since it is not obvious that the galaxy has already been colonized by other civilizations, there may be a Great Filter stopping this from happening. Miller uses the analogy of a person about to climb a mountain, believing that everyone else who has attempted it has died in the process.

Several challenges were discussed by attendees. (If you attended the lecture and want to claim credit for your question, please comment below, thanks.)

1. It's too early to say there are no civilizations; it may not be so easy to detect them or rule them out. We're still discovering metazoans in Manhattan so it seems a little early to rule out von Neumann probes on low gravity bodies in the solar system. We've barely begun to catalog the fauna of our own ocean floors. We could not detect a twin Earth emitting the same radio energy (the C-index), even if it was orbiting Alpha Centauri. Miller points out that even if there were only a few civilizations in the Milky Way preceding us, "the galaxy is older than it is big", and these earlier civilizations could have colonized it already.
   
   
2. He made the point that the things which prove advantageous in the midst of evolving on a single planet might have no such advantages in terms of galactic colonization. Very true; I would argue that we are much more likely to find alien artifacts, than the aliens themselves, as all of us meat-creatures might be stuck on our planets while our machines colonize the galaxy. To that end, (my point) it's entirely plausible that the Solar System could be littered with space probes and we haven't found any yet, or did, and just didn't know what we were looking at.
   
   
3. I would therefore extend Miller's analogy like this. Only in the process of climbing the mountain, does our climber develop wilderness skills and begin to see things that resemble his own boot tracks, etc. and finally as he approaches the summit realizes that lots of people have climbed it, come down the other side, and their descendants have built large villages which due to his previous ignorance he has not been able to locate. (Or, maybe just some of their livestock, trained birds-of-prey, etc. have made it.)
   
   
4. Active attempts to bring ourselves to the attention of aliens have occurred (METI) and been roundly criticized. Miller notes that the risk of extinction from aliens over the next few centuries is lower than eg bio-terrorism or an intelligence singularity. True; but we still may be making life more difficult for our descendants. Related to this, he proposes an ingenious experiment that for a month we should shout our heads off electromagnetically, and see if there is any strange activity. While I agree it's unlikely we'll get invaded next week, I still think the risk:benefit does not work out and there are just too many unknowns, and we may be screwing our distant descendants. Miller suggested that enforcing a moratorium on METI-like activities is probably impossible.
   
   
5. He argues that technological singularities of the paperclip maximizer variety are unlikely to be a major contributor to the Great Filter, because we would be able to see the boundary of it as it expanded (unless it was doing so at light speed.) My concern with this is that, while an AGI might be much smarter than its creators, it is still not omniscient, and the impact of its actions could in principle still outstrip its ability to predict that impact. This is the story behind the rise of human intelligence and the sixth great extinction that we're living through, but has happened in pulses of endogenous extinctions throughout Earth's history (the rise of superpredators every fifty million years or so, the Oxygen Catastrophe). The lesson of evolution here on Earth is that the smarter things are, the faster their behavioral plasticity "catches up with them" in exactly these sorts of disasters, so to suppose that alien paperclip maximizers are immune to this problem is to argue that a qualitative change in ecological dynamics has occurred.
   
   
6. There were two (possibly unappreciated) related questions asked: one about civilization perhaps being bad for sustaining civilization (witness declining birth rates in the developed world) and another that intelligences might prefer virtual reality - involution - to expanding into space. Miller points out the passive version of the "baseball bat" problem: you can live in heaven, but if a bad guy comes and bashes your server with a club and you as you sleep in your VR pod, that's the end of it. (Related: dynamic complex systems like minds, in principle, tend to drift toward delusion and suffer inherent cyclic crises.) It's a thesis for someone in psychology or a related field to note whether there is causation or just correlation between the increasingly encompassing virtual reality-like entertainments available in the developing world, and declining birth rates.
   
   
7. One questioner asked about the distinction between intelligence and civilization - humans have had a "civilization" only since agriculture. This was a really original line of thought. Therefore, there could be many alien intelligences, but few or no civilizations. One solution for humans avoiding the Great Filter would be to abandon civilization and go back to hunting-gathering - not directly suggested, but this is the only implication of such an argument I could think of. The extreme number of assumptions built in to discussion of alien civilizations should always be pointed out - civilization is something that collections of human nervous systems do, and it is not clear it is a necessary consequence of intelligence. (As a physician I ask: do we assume the aliens will have similar EKG waveforms and liver enzymes as us? No, because that's ridiculous. So we do we assume that the even more complex activity of another organ, that we don't even share with other animals on this planet, is automatically going to be meaningfully similar?)

There's also a psychological point to be made about "big picture" arguments (the singularity, the Fermi paradox, the simulation argument, etc.) They have a tendency to converge on either prophetic religion-like conclusions (e.g. the singularity as the rapture for nerds) or Lovecraft (the estivation hypothesis, which was mentioned in a question and made me think about this.) When we talk about these things, there are many many unknowns. In such discussions, I think there is a tendency for the resulting arguments to resemble the internal contours of the human mind, more than any future events in the actual external world; hence their regression to religion-like conclusions. This does not mean such an argument must be incorrect, but it should make us suspicious when a big-picture argument hews too close to our "ontological test pattern. "

Consider in contrast cosmologists' models of the distant future of the universe, which concern physical objects which we can now observe and characterize, using rigorous mathematical rules. These models often seem boring, meaningless, difficult to understand, and unsatisfying. This is exactly how we should expect most models will seem of things outside our own and our ancestors' experiences, or beyond the scale of time and space to which we are accustomed and which we are built to perceive; the further outside their experience, the moreso. This occurred to me when we were discussing the estivation hypothesis, though overall Miller's arguments do not set off many alarm bells for this quick-and-very-dirty heuristic.

The Earth Has Not Been Disassembled for Computation - Percent Utilization of Phosphorus and Nitrogen on Earth by Living Things

A 2015 paper by Landenmark et al estimates the total number of DNA bases in nature as 5.3x10^31 megabases. This of course leads to questions like: how much of the elements on Earth is life on Earth using? I'm aiming for an answer within an order of magnitude. This has implications for concerns about AI takeoff that I will return to at the end.


NITROGEN

Living things occupy slightly more than a billionth of the planet's nitrogen in our DNA (0.000000115%). Living things occupy 0.0023% of the planet's nitrogen overall, the lion's share of which of course is in protein. (See my assumptions below if you like.)


PHOSPHORUS

Living things are using only 0.00047% of the planet's phosphorus in our DNA - but that expands to 4.7% of the planet's phosphorus in living cells overall. This is a much more significant fraction.


Does this difference exist because life on Earth has chosen phosphorus as, effectively, energy currency to manipulate gradients? Or because nitrogen is harder to make biologically available? Even now we rely on relatively few bottlenecks to fix it.



IMPLICATIONS FOR AI TAKEOFF

There's no reason to assume that these numbers represent a global, rather than local optimum for resource utilization for replicators on Earth. That said, we've had four billion years to optimize. This is relevant because of the concern that AI taking off without regard to human welfare would disassemble the Earth into atoms for computation - the farther we are from truly optimized resource utilization, the more an intelligence explosion would be disruptive to the status quo. I found the Bar-On paper on amount of DNA in the biosphere from a link in a discussion about the computational efficiency of nucleic acids in cells. The latter paper suggests that protein translation is several orders of magnitude faster than the fastest current computers, and only an order of magnitude under the Laundauer limit. Of course, resource utilization and computing speed are two different variables, but it seems computation is getting near optimized already - and yet, no disassembly of the Earth for phosphorus. Not even 5% of the energy currency atoms are put to work! Of course, an AI would be qualitatively and quantitatively different in unpredictable ways from what came before, in which case there is no point in discussing this - but the replicators that exist in reality make the best starting point for such a discussiong.

What's more, protein translation is computation in the service of replication. It is quite likely that AIs would end up being selected in much the same way as cells have, with limited resources to be dedicated to refining the model of the universe (getting smarter.) The ivory tower AI super-minds would be dominated by the silicon bacteria. Of course, this is still no reason to think a hard AI takeoff could be disastrous for all life on Earth, an extinction like we've never seen - which the AIs themselves might not have the foresight to survive - but if they do, the best bet is that they will "revert to the mean" of all replicators, with making copies as the goal.




An imperfect analogy. In nature, you have to make do with what's there. The shapes aren't friendly for efficient packing and there are a lot more holes.

Assumptions:

I could not find estimates of the overall mass of nitrogen and phosphorus in the biosphere, so I used the percentage weights in living cells, and derived from a paper estimating the mass of carbon in the biosphere at 5.5x10^14 kg (Bar-On et al 2018), along with carbon being 18% of the atoms in living things.

For both I used 2884.6 kg/m^3 mass of the Earth's crust (weighted the differently dense continental and oceanic crusts at 0.3 and 0.7 resp.) My number for nitrogen comes from nitrogen in the atmosphere, plus nitrogen in the top meter of the Earth's crust, estimating mass of the atmosphere as 5.15*10^18 kg, of which 78.09% is nitrogen, and abundance in the crust as 0.002% by mass (there was some conflict over this between sources actually of up to an order of magnitude; but there is so little nitrogen in the crust compared to the atmosphere, about 347,000 times less using this number, that it's still a rounding error. I assume that there are an equal number of A T C and G which means 3.75xnitrogen atoms per base.

For phosphorus, I used a crustal abundance of 0.1% mass, ignoring the negligible phosphorus in the atmosphere. There is 1xphosphorus atom per base. The major "slop" in this figure occurs because different organisms have different fractions of phosphorus, for one thing since phosphorus is used in structural molecules like bone (85% of phosphorus in humans is in bone; even the same organism at different ages differs substantially, e.g. 0.5% in infants, close to 1% in adults.) Bacteria come in at 0.9% (3% dry weight, assuming 70% water mass per cell) so I used that figure, since bacteria outweigh us by a factor of a thousand, and the number is intermediate even for the values for vertebrates.


REFERENCES

Bar-On YM, Phillips R, Milo R. The biomass distribution on Earth. PNAS June 19, 2018 115 (25) 6506-6511.

P. Kempes CP, Wolpert D, Cohen Z, Pérez-Mercader J. The thermodynamic efficiency of computations made in cells across the range of life. Philos Trans A Math Phys Eng Sci. 2017 Dec 28; 375(2109): 20160343.

Landenmark HKE, Forgan DH, Cockell CS. An Estimate of the Total DNA in the Biosphere. PLoS Biol. 2015 Jun; 13(6): e1002168. Published online 2015 Jun 11. doi: 10.1371/journal.pbio.1002168

Michael Schirber. Chemistry of Life: The Human Body. Livescience.com. https://www.livescience.com/3505-chemistry-life-human-body.html#:~:text=Oxygen%20(65%25)%20and%20hydrogen,%25)%20is%20synonymous%20with%20life.

The Singularity Will Be An Extinction Event, and an Endogenous One

There have been exogenous extinctions, ie not from an ecosystem's "internal contradictions." Examples are massive magma flows like the Central Atlantic Magmatic Province at the Triassic-Jurassic boundary, or the asteroid strike like the K/T Boundary. These were at least partly caused by out-of-context events that life on Earth did not influence. Then there are endogenous extinctions, which were caused entirely by the actions of the system itself, with no external disturbance. The best example is the Great Oxygenation Event, where the cyanobacteria inadvertently poisoned themselves, and paved the way for a whole new kind of metabolism. About every 26 million years, a superpredator develops and kills everything – humans are filling this role currently – and even if there's not an extinction, there's a local minimum in biodiversity and ecological robustness.

Since we're the aerobic beneficiaries of the Great Oxygenation, we like to narrativize this in the form of a teleologic happy ending. That is: the story becomes, yes the cyanobacteria poisoned themselves, but it was to make way for the glory of oxygen-breathing life. That oxygen they fatally polluted themselves with turned out to be an improvement, a new fitness landscape. Any endogenous extinction clears the way for evolutionary progress!

This is false. Of course the Great Oxygenation Event turned out to be survivable, because we're here looking back on it. But choose any other model example of a closed ecosystem where the endogenous activity of the local organisms is rapidly changing their environment, and you are unlikely to find that the majority of them are success stories. Things poison themselves, and end up with no descendants that can survive. (There is no argument to exclude humans from this phenomenon. Both deforesting Easter Island and the ongoing Great Carbonization Event are good examples.)


Two implications follow:

1. The reason for the Great Silence (ie the Fermi paradox) could be that there are many watery worlds out there which evolve local cyanobacteria, but they have their own endogenous shocks, and these do not result in a survivable planet, or at least in a richer potential fitness landscape. As in Conway's Game of Life, if they're lucky they either settle into a simple oscillating system (bloom, mass extinction, bloom, same kind of mass extinction, ad infinitum) or the ecosystem collapses completely and ends.

Speculation regarding this: we're fairly confident the first metabolism on Earth was sea vent iron sulfur organisms, using sulfur in what is now oxygen's chemical role. The Great Oxygenation may have only happened when it did, a full 1.5 billion years after the first life and at least 800 million years after photosynthesis appeared, because an asteroid delivered molybdenum, allowing nitrogen fixation and more efficient anaerobic metabolism. Whatever the reason, had this happened prior to photosynthesis, we may have ended up with an Earth poisoned with sulfur or at least with a massive amount of oxidized sulfur.

In an interesting parallel observation: we're also confident that Venus was once a wetter, cooler world that had a runaway greenhouse effect. One of the mysteries of Venus is the origin of all the sulfur in its thick atmosphere; to a first approximation all sulfur on Earth's surface is assumed to be from volcanoes, but why so much more on Venus? Another mystery is the identity of the small UV absorbers (about the size of bacteria) that form the dark bands in its atmosphere; one idea is that they're cells descended from ancestors that evolved at the surface and now can only survive in the more benign lower temperatures and pressures of the high clouds. If indeed these are the survivors of a Great Sulfuration Event, while the event did not result in total extinction, it limited the Venusian ecosystem to oscillate on a barren fitness landscape, just from the bad luck of having richer crust contents or earlier impacts with potential-enzyme-cofactor-bearing asteroids that allowed more efficient iron-sulfur metabolism.

(Recent evidence however suggests a massive volcanic event 700 MA ago that resurfaced the planet after massive flows; this which may be enough to explain all the sulfur. A gradual boil off of water remains quite likely, for two reasons – the D/H ratio on Venus is about 150 times higher than Earth, where comets have at most a 3 times higher ratio than Earth, suggesting loss to space of hydrogen from water and preferential retention of the heavier nucleus; and that such a massive volcanic event could have been caused by the loss of water, and the cessation of plate tectonics which allow a cataclysmic buildup of heat. It's interesting that the Siberian trap flows and CAMP happened during a period on Earth when the continents were crammed together and perhaps less efficient at letting out volcanic heat, though these events were still nowhere near what happened on Venus.)

2. If a technological Singularity occurs, it would be an endogenous extinction. In this case we are the cyanobacteria, and our extrasomatic adaptations are the contradiction internal to the system, and the AIs are our oxygen-breathing descendants. Like them, we produced the conditions that destroyed us and paved the way for the next phase of life. It's true that cyanobacteria and anaerobic organisms persist but do not dominate the world as they did in the Archaean. Even if cellular life survives the Singularity, being relegated to the role of cyanobacteria is unappealing for most.

But then there is another possibility, in which the AIs drive themselves extinct too. Think of this as the super-pessimistic case. Singularity optimists think we can benefit from or at least co-exist with superintelligence (becoming the equivalent of cyanobacteria is actually optimistic in this scheme.) Singularity pessimists think the event will kill all biology. Here, I suggest the super-pessimist position, which is that the Singularity may kill us, then also itself, in the final, most spectacular ecocide of Earth's history. Why? One theory is that any self-improving superintelligences will necessarily disassemble matter, including whole planets, into atoms that can be used for computation. But there is no principle stating that intelligence must always exceed power; that is, that impact of behavior must grow more slowly than ability to predict impact of behavior. Certainly it didn't happen with cyanobacteria, and given the sluggishness of our response to global warming it might not be happening with humans. Even if the AIs are in fact superintelligences, they are still not omniscient. As they're disassembling everything, they may get to the end of a predictive computation and realize that part of the code has gone cancerous and is replicating out of control (and consuming matter in the process) and can't be called back, or they're going to run out of power before they get to the next planet or star system, or overheat, or whatever problem an AI might run into.

Therefore, if the Singularity does happen, it would be just one type of endogenous extinction. If in a hundred million years, aliens or their self-replicating probes visit the solar system (if such things ever occur in the history of the universe) they might find its dusty, partly-disassembled remains, and file the data under "ecosystems that ended with behavioral/artifactual singularities" and then move on. Interestingly, we have already found old planetary systems that are far dustier than we would expect, with no explanation for the inner dust ring and a some constant replenishment process. Even this assumes that the self-replicating alien probes can get there before becoming cancerous dead-ends themselves.

An Existential Risk Comparable to the Singularity

Imagine that due to some breakthrough in computer science, we can determine (with certainty) that for a certain architecture, there are some designs that will "work"; that is, when activated, become a successful recursively self-improving general AI. Now assume that 13 such designs have already been tried. A tiny fraction to be sure; and yet in no case has there been any discussion before the research team turned each of them on, no input from other researchers, no public comment, and certainly no attention from policy makers of any sort.

The catch is that this architecture is highly iterative, and it has to run for a long time before you find out if it's going to "wake up" - consequently the researchers load the software and hardware into satellites, because the earliest any of them would "wake up" would be 2036. (For our purposes, assume once launched, these satellites are out of reach - like Elon Musk's car.)

I assume the AI safety community would have something to say about this; about people unilaterally turning on instances of this architecture, and placing it out of reach. Unlikely though it is, any one of those could wake up and we could find the Solar System transformed overnight, and not necessarily to humanity's benefit.

Why such an esoteric thought experiment? It's not really a thought experiment. There have already been 13 active attempts so far (that we know about) to signal nearby star systems. Given the constraint of the speed of light, the earliest we could hear back (or meet someone/something) from any of them would be 2036. Much like the satellite-launched AIs, once you send the message, you can't delete it from their inbox. The 1-in-29 million comes from the original Drake equation estimate of 3,500 civilizations in the galaxy, and one hundred billion star systems. Note that this thought experiment assumes every species is confined to one solar system, but if they have interstellar travel (and follow the signal back to the source) then that 1-in-29 million probability would be much higher.

AIs are at least designed by humans, with possible ethical constraints. Aliens able to visit our solar system would not in any way have our interests at heart. If you're in the rationalist community and you're concerned about a technological singularity, you should be very concerned about existential-risk-level-dangerous wildcat attempts to reveal our presence to other solar systems. This is called METI (Messaging Extraterrestrial Intelligence) instead of SETI, and you can read more about stopping it here.

AIs Are Beating Experts at Old, Relevant, Real-World Problems

When I was a molecular biology undergrad in the mid 90s, the Big Question in biochemistry was protein folding - how to predict the 3D structure of a protein from its primary (linear) sequence of amino acids. Solving this problem would provide a massive boon to many fields, not least of which is drug development. But it's a hard problem: a factoid cited to impress this upon people is that if proteins folded by passing through every possible configuration randomly until they found its working (not usually lowest energy) conformation, the process would take longer than the lifespan of the universe so far. Obviously this isn't what's happening. And yet, the promise of number-crunching power being used to finally solve this always seemed just a few years away - many people, in retrospect comically, thought the problem would be solved by the close of the century.

The problem is still not solved, but scientists have made incremental progress, and now, the closest thing to a quantum leap forward. Every two years there is a contest where scientists competitively try to solve a structure, then get together and see who got closest. Mohammed Al Quraishi is a computational biologist at Harvard who writes about the 2019 meeting, and who the winner was: an AI named Alpha Fold.

This is exciting because computational biology may really be about to start paying dividends. It's scary because it's one more place where AI is starting to automate our jobs, even Harvard professors. Al Quraishi makes many interesting observations in his post, among them, that a technology company showed up and ate the protein chemists' lunch, and the reaction was muted; we're all becoming numb to this sort of thing. He tries to wake us up by asking how academic computer scientists would react if at a conference, a pharmaceutical company's scientists had showed up and beaten them at their own game.

I had previously made the informal argument that AIs would naturally exceed human performance on games, as games are clearly defined processes with discrete rules and entities, exactly the sort of thing that computers are good at. The difficulty comes when machines must interact between the information-overloaded not-fully-understood messy real world, which is why (again, I informally argue) computers are great at parsing text, and even writing text in the style of sports journalism or certain authors - but at bottom, these are all just very complicated echo chambers, with no meaning or subjective experience associated with the words and sentences. And indeed, the language manipulation has come first, but that doesn't mean encoding external experience in language is impossible. I had maybe subconsciously thought that something like protein folding, a perfect example of a messy real-world process, would be beyond the capabilities of machines at least for many years, but this theory has now been clearly falsified.

STOP METI

tl;dr if you worry about the singularity or bio-terrorism as an existential threat, you should help the effort to stop efforts to announce our presence to aliens. This should be a priority for the rationalist and effective altruism community.

Many people are familiar with the SETI project, Searching for Extraterrestrial Intelligence. METI stands for Messaging Extraterrestrial Intelligence. Stephen Hawking, Elon Musk, and Freeman Dyson believe this is incredibly stupid and dangerous.

Of the 13 (known) attempts to deliberately signal another star so far, the earliest that any response could be received is 2036. The earliest that any response could be received from a sun-like star with planets in the habitable zone is 2085. Additional attempts are almost certain to be made in the next few years.

Most scientists recognize how important - in fact, world-shattering - contact with aliens would be, and there's actually a protocol for what scientists should do if a message is received. But these attempts are being made by individuals or small groups, with no oversight, often with really stupid justifications. One was an art project; this one invited kids to compose the message.

The argument against deliberate messaging is that even here on Earth, contact between members of the same species with differing technology was catastrophic for not just the humans on one side, but the ecosystems. A visit from aliens with enough technology to detect us or visit would therefore likely be devastating, even if they don't have malicious intentions. Once we're detected, we can never be un-detected.

The arguments for METI are laughable, and best thought of in terms of a native American on the shores of the Atlantic, talking about building signal fires to bring the Europeans over even sooner. Their best arguments are:

• Aliens might already have noticed us anyway. (So why make it more likely?)
• It's extremely unlikely anyone will get the message (So why do it at all?)
• They won't come for a long time. (If we discovered a form of energy that would start poisoning our descendants in 10,000 years, would we use it?)
• If all the other aliens are remaining silent, then by doing the same, we're part of the problem, and we're hypocrites for trying to detect others (given the risk:benefit, that's a trade most of us would be comfortable making.)
• Aliens who can respond or come here will necessarily be moral beings and won't hurt us. (This one is really absurd, and not only makes assumptions about the intentions of the aliens, it sounds very much like a religious conviction.)

You can see a more thorough treatment of these arguments by a SETI expert in this paper, and the abstract finishes with these sentences: "Arguments in favor of METI are reviewed. It is concluded that METI is unwise, unscientific, potentially catastrophic, and unethical."

You can see some of these arguments being made with a straight face by METI's founder Doug Vakoch in this article. Note that METI is a splinter of SETI, since most of the scientists involved in SETI forbade active communication attempts.

Going forward I'm going to do my best to create awareness in the rationalist community and put priority on this as an existential threat alongside AI. My plan is to contact the people at SETI to see what they're already doing and how others can contribute. It seems like the best approach for now is stopping transmissions by blocking individual projects, but ideally there would actually be a law against this as well as norms that socially punish defectors. To end on an optimistic note: because there is a chokepoint (money and limited time on transmitters which are mostly controlled by universities) this problem is actually much more tractable than avoiding a "bad hard takeoff" of general AI.

The Worst Argument Against Being Cautious About AI

Note: throughout this article when I use the term "AI" I'm talking about superhuman artificial intelligence, and I'm assuming the counterarguments are referring to same.

There is a species of article or blog post that basically boils down to "People who worry about AI as an existential risk are being irrational for reasons the follies of human psychology, and/or overarching principle linking intelligence and moral behavior." (I won't bother linking to one, you can do a quick web search and fine a lot of them.) In other words, they're attacking a straw-man argument that states: the reason to worry about AI is that it will necessarily be malevolent toward humans.

A side comment often appended to these arguments make is that to believe AI will be malevolent toward humans is to be comically anthrocentric. While this observation is quite correct, misunderstanding this point is exactly why these arguments are so wrong. You have to be comically anthrocentric to think that we are immune from the disruptions caused by superintelligence. That is to say: for AI to hurt us, it doesn't have to come after us specifically. It just has to not specifically care about us.

Case in point: settlers from Europe in the Americas were not malevolent toward the native humans, for the most part. They just found arable land that they wanted, and the didn't especially care who was already there - resulting in plague and death and the loss of many cultures and languages. (That was just between two flavors of humans where one side had about a four thousand year technological head start.) In the same way, the developers at the edge of my city don't have anything special against the deer and coyotes and worms in the soil - but the spreading pavement and light and traffic and noise has affected them just the same.

To repeat: for AI to hurt us, all that has to be true is that it does not care about us. It doesn't even have to have any special malevolent intentions.

There are projects that are essentially trying to solve morality before the first AI goes online, for exactly this reason. I wish them luck, but we've had a few dozen centuries on this project and not gotten very far.

General AI: Computation versus Survival, Superintelligent is Not Omniscient

It is usually assumed that a superintelligent AI would maniacally focus on improving computation. Just to highlight the centrality of computation, a recent paper in the British Interplanetary Society Journal argued that the reason we don't see aliens is they're sleeping, waiting for a time when the universe is cool enough that their computations are more efficient. The alien singularities are waiting until they don't need to be cooled.

The most common concern associated with this line of thinking is that the technological singularity would be bad because the AIs would use all available resources - starting with all matter on Earth, including us - as computational resources. While I think a technological singularity would be catastrophic, I think the reason is eve more mundane.

Of course, this assumes that the AIs in all their power are maximizing computation. I don't think this is questioned nearly enough, and a good bit of the inertia around it stems from the cultural assumptions of the programmers and engineers making the argument. The singularity is thought of as a logical outcome of Moore's law, which concerns exponential growth in computation. It's not clear that this is what an AI would necessarily be maximizing. For our part, humans and other animals maximize a host of confused and often contradictory goals. Of course we remain in this mess because we are not recursively self-modifying. Assuming that AIs with such an ability aren't automatically condemned to wirehead, it's not unreasonable to ask whether there are things to maximize that increasing computations just wouldn't fix.

Replicators whose descendants are present into the future are the result of selection for one thing - making copies - and to the extent that extra computation can improve that, then the AIs present in the future will be selecting for computation that helps them reproduce and sustain themselves. But even a superintelligent AI is not an omniscient AI, and cannot see infinitely into the future and understand ahead of time the impact of all its actions in maximizing its survival OR computation. My strong suspicion is that a hard takeoff will likely be an apocalyptic gray goo explosion, much more thorough and faster than the mass extinctions so far in the much more comparatively mildly ecocidal anthropocene, and that furthermore this is a strong candidate for the Great Filter and the Fermi paradox. That is to say, we're more likely to find the simple but fecund survivors of such an event as something that looks like post-singularity AI-algae (or free-roaming AI "cancer") than alien AIs that are interested in philosophy.

Russian Short on Flying War Machines

Normally I care more about the ideas than how they're rendered, but this is outstanding enough that it made even my blunted retinas happy. The theme is somewhere between the Terminator and Bradbury's There Will Come Soft Rains.

Крепость/Fortress from Dima Fedotof on Vimeo.

Review: Progress Toward the Singularity

This focuses on ways in which GAIs could interact with the material world, rather than GAI itself. In either domain I make no guarantee of currency; you will note that most of these links are 2-3 years old and have presumably been developed and found application since then.

1) Automated lip reading. It's not just HAL these days.

2) Remotely reading the position of people in a room based on how our bodies distort WiFi fields.

3) Smart sand.



4) Drones through your mail slot.



5) Computers that can spot microexpressions and lies better than humans, as well as microchanges generally (color and motion changes).

6) Stock trading programs doing things that we can't understand, faster than we could even if we did - termed by researchers in Nature a "new machine ecology" of software.

7) Generation of mathematical proofs that no human understands, but which can be shown mechanically must be correct. P = NP? If we can't understand it, then for human brains, NP doesn't even equal NP!

8) Reading your mind and knowing what you're looking at; they can reproduce images from movies as you're seeing them.

9) Forget chess. They can beat everybody at rock paper scissors too, because you're a lot more predictable than you know (but not than the computer knows). The best you can do is flip coins and come to a draw.

10) A fair amount of sports print journalism has been replaced by computer-written articles already.

For now, all these technologies are very much dependent on large amounts of money and attention from humans, and are in no way self-replicating. But we should expect that those places where the most utility is derived by humans are where we see powerful (read: expensive) new technology first put into place - like the stock market, and warfare. Ask Al Qaeda, who have been getting gradually exterminated by the HKs circling over Afghanistan for over a decade. (If we don't call the first truly autonomous drones HKs, that would represent a missed opportunity.)

Skynet's Ancestors: Are Groups of Humans Also Conscious?

I just re-watched Terminator 2, which is aging pretty well, not least because it used CGI as opposed to live action latex-and-juice effects in the first one that today look fairly bad. A refresher: Skynet sends a new, scarier shape-shifting terminator back in time, this time to kill John Connor as a 12 year-old instead of killing his mother as in the first one. To counter Skynet, the grown-up John Connor sends back a re-programmed terminator to protect him. It turns out that there were parts recovered from the first terminator from 1984, and a tech company has been reverse-engineering them - creating a loop where Skynet essentially catalyzes its own creation. A big part of the movie is the side-mission taken by the Connor family and their pet Austrian to destroy the lab working with the old terminator's parts, thus stopping the creation of Skynet.

Of course this plot was an excuse to get Schwarzenegger in another movie even though the terminator (well, the first terminator) was killed in the first one, and to make him a much more central character. But the most interesting thing to me isn't that, or even the paradox of a self-causing AI. Certainly the idea of fate figures prominently in the second and third movies, and many who think the Singularity concept is a coherent one also think that such an event is inevitable, given a technological civilization with enough time.

And that's why the most interesting thing to me about T2 is the police. (How many? "All of 'em, I think," says the young John Connor.) First of all, the parallelism is interesting: fine Skynet, you try to kill John Connor's mother? Well then we'll kill your mother, what do you think of that? (As a side note, because the shapeshifting Terminator made no attempt to defend Cyberdyne, either Skynet didn't think of this, or had already seen Terminator 3 and knew it didn't matter.)

1970's Colossus, which deserves to be more clearly remembered as introducing us to the idea of defense computers using the nuclear weapons under its command to sinister ends. (Although if you were Skynet, you could do better than something so obvious as a nuclear war.)



But much more fascinating than that: even though there is no fully functioning, self-perpetuating AI inside Cyberdyne systems, a more dilute but still discrete pattern has begun forming around the primitive Skynet's birthplace, a wealthy society's resources marshalled to protect the womb. And much like in our own tissues, Cyberdyne's individual T-cells (investors, police) cannot possibly appreciate the ultimate result of the developmental process they're taking part in. It's hard not to see the invisible hand of Skynet's intelligence, already embedded in the logic of the human scientific enterprise and capital accumulation and the institutions we've created to advance those processes.

To be clear, I don't get the impression that Skynet was somehow magically reaching back through time and hypnotizing LA's finest (and Cyberdyne's investors) to protect its primordial ancestors, or that this is what James Cameron was suggesting. But it's interesting to contemplate not just in light of the concept of the Singularity, but of the serious philosophical debate about the basis of consciousness, and to what kinds of entities it can apply. If you think consciousness comes from a pile of neurons being connected, then does it matter whether some of the neurons are inside separate skulls, connected by speech rather than synapses? That is to say, if you believe that one person is a conscious entity, doesn't that mean two people must form an additional conscious entity, as would any combination of humans? So things like Wyoming, and Honda Motors, and investors in the stock market, are conscious entities! As would be Cyberdyne Systems (and its associated financial and security apparatus) - and Skynet is the eventual expression of the converging currents of that particular self-organizing system.

Trolley Problems Are Now Real Problems

Cross-posted to Cognition and Evolution.

Trolley problems are moral exercises (or experiments) where participants are asked to choose between horrible choices, to see how our moral reasoning works. For instance: do you push one person onto the tracks of an out of control trolley to keep five people further away from getting run over? Or do you just not say anything if you see a single careless person wandering into the trolley's path of his own accord, to keep those five people safe? Among other interesting observations in these experiments, we are inconsistent, and people who claim vastly differing moral foundations (religious and atheists, for example) tend to choose the same answer, as long as they're otherwise from the same cultural background (e.g., they're both American). Some enterprises already exist attempting to "solve" morality, that is to be able to program it into a computer, partly motivated by a belief in the impending technological singularity.

Trolley problems are criticized as being able to give us a window into actual moral reasoning for some of the same reasons same reasons as any consequence-less self-report method. But literal trolley problems are now becoming consequential, now that we have driverless cars. Their engineers have to encode how these cars will decide who to hit, if they find themselves in a situation where they have to hit someone. If they have a choice between a helmeted and non-helmeted cyclist, shouldn't they aim for the helmeted cyclist to minimize harm?

If only things were as simple as Asimov's Three Laws of Robotics.

"An Ethical Governor for Constraining Lethal Action in an Autonomous System"

Paper by Arkin, Ulam and Duncan at Georgia Tech, intended for use by U.S. military hardware. Three Laws of Robotics? Is this what MIRI would consider friendly AI? Where is theirs?

Time Travel Is Not Possible or Is Near-Impossible to Detect With Current Methods

Time travel is fun to think about, in part because the argument structures involving its effect on the world (if it were real) appear in other kinds of discussions. I'm posting this now because of the fun study that Nemiroff and Wilson did, looking for evidence of time travelers by finding evidence online of people making references to events before they occur. They did not find any such evidence; publication here.

To address the burning public interest in their project, they did an AMA on Reddit, and when asked "Did you receive federal funding for this endeavor?" they responded "Yes. We used funds left over from our study titled: 'Does Tax Payer Money Burn Any Better Than Regular Money?'" (They really did say that but went on to clarify that the real answer was no.)

Let's say time travel in the Back to the Future sense is real. Say that if you go back to the past you can change the present to which you return (no boring branching timeline stuff.) And say that if you remove something from the past, it disappears until you carry it to the present. That is: imagine you have sentimentally precious heirloom silverware. No you don't! I took it from your grandmother when she was a young hottie, and when I get out of the time machine to give it back to you today, you look blankly at me and say "Why are you giving me old forks?"

If that's how it works, shouldn't we expect commodity runs on everything? That is: you realize that gold is valuable. You go back to the pleistocene before people valued it or knew how to pull it out of the ground. You bring back a whole consortium of miners and investors with you, running around the world pulling 50% of the shiny yellow stuff out of the ground, all to the bewilderment of frightened hominids, hiding behind rocks as they watch the incomprehensible doings of these new hairless creatures. You come back to the present and - bammo! Yes the value of gold falls in half overnight, but you still have 50% of that. 25% of the total previous gold market is still nice.

No you don't! Because unless time travel dies with you and your investors, people from YOUR future will sabotage you and take it for themselves. And from their future...and their future...etc. What this means, in economic terms, is that if this kind of time travel is real, then of course it's already happened, and people furthest in the future have perfect information about everything (100% efficient markets) which means everyone has it. Or, that no one has it, because there's no stable reality. (Remember when Bill and Ted said "remember to do X in the future" and got out of a jam? Why can't their enemies say "remember to stop Bill and Ted from doing X"? And so on and so forth ad infinitum. Neither Bill and Ted being the good guys or writers not being able to think their way out of a plot problem can count for your answer.)

So we have either
a) a totally efficient market with everyone knowing the allocation and value of all items throughout time
b) a constant maelstrom of shifting reality

If it's a., the fact that we still see things with apparent value around us instead of in the Big Commodities Exchange at the End of Time (or that we're not getting invaded by people from the End of Time looking for causal high ground) means either that everything "has already happened in order" (the simple Douglas Adams model of time travel), or that time travel of this sort is not possible, or that nothing has any value in a 100% eternally efficient market. (It also means no free will, because we're basically all just watching ourselves in a movie then.) And here's where things start to track other arguments. Here we run into an argument familiar to Singularity buffs, simulation argument maniacs, and Fermi paradox jocks. And it's basically this:

X appears to be an arbitrarily powerful process not provincial to humans. We look around the universe and we do not see evidence of X dominating everything. Therefore, either X is not powerful, or we can't see it, either because it's not see-able (to us), or we don't know what to look for.

Fair enough, except it's very difficult to tell the difference between "it's not see-able (to us)" and "I'm a snake oil salesman insisting that you believe me without evidence". That's also called a PEP, a Pointless Epistemological Problem (see #3 here). Fans of politics, religion and marketing will find these sorts of claims familiar.

Returning to time travel for a moment: the fact that aliens have not used time travel (rather than measly space travel) to visit us (and beat us up!) suggests (again) some combination of: there are no intelligent aliens; we are of little interest to future galactic events because we are boring, weak, or go extinct soon; or that everyone is nice to each other forever. Boy, talk about unrealistic!

The Burgess Shale of the Singularity

This is the house in Palo Alto (or rather, the detached garage) where Bill Hewlett and Dave Packard built their oscillator, HP's first successful device, in 1939. Although it's a private home it's listed as a landmark, and while I was taking this picture two other groups of people came by to pay homage (one from France, one from Brazil) which made me feel not so weird for walking over there.


When the machines are looking back at their own Cambrian explosion-equivalent, perhaps they will view this place as the primordial pool from whence they arose.

The First Amazon Drone Delivery Should Be To Sarah Connor

Abrupt Rise of New Machine Ecology Beyond Human Response Time

That's a new paper about ultra-high frequency trading in Nature Scientific Reports by Johnson et al. Link here.

Singularity Watch: Your Movements Detected By Existing WiFi Technology

Paper here. Now HAL can see you picking your nose. And if you haven't seen this, give it a second, it's worth it:

Skynet is Already Practicing on Pigs

Wild pigs are being exterminated with drones. At least the apocalypse will have bacon.