The Earth Has Not Been Disassembled for Computation - Percent Utilization of Phosphorus and Nitrogen on Earth by Living Things

A 2015 paper by Landenmark et al estimates the total number of DNA bases in nature as 5.3x10^31 megabases. This of course leads to questions like: how much of the elements on Earth is life on Earth using? I'm aiming for an answer within an order of magnitude. This has implications for concerns about AI takeoff that I will return to at the end.


NITROGEN

Living things occupy slightly more than a billionth of the planet's nitrogen in our DNA (0.000000115%). Living things occupy 0.0023% of the planet's nitrogen overall, the lion's share of which of course is in protein. (See my assumptions below if you like.)


PHOSPHORUS

Living things are using only 0.00047% of the planet's phosphorus in our DNA - but that expands to 4.7% of the planet's phosphorus in living cells overall. This is a much more significant fraction.


Does this difference exist because life on Earth has chosen phosphorus as, effectively, energy currency to manipulate gradients? Or because nitrogen is harder to make biologically available? Even now we rely on relatively few bottlenecks to fix it.



IMPLICATIONS FOR AI TAKEOFF

There's no reason to assume that these numbers represent a global, rather than local optimum for resource utilization for replicators on Earth. That said, we've had four billion years to optimize. This is relevant because of the concern that AI taking off without regard to human welfare would disassemble the Earth into atoms for computation - the farther we are from truly optimized resource utilization, the more an intelligence explosion would be disruptive to the status quo. I found the Bar-On paper on amount of DNA in the biosphere from a link in a discussion about the computational efficiency of nucleic acids in cells. The latter paper suggests that protein translation is several orders of magnitude faster than the fastest current computers, and only an order of magnitude under the Laundauer limit. Of course, resource utilization and computing speed are two different variables, but it seems computation is getting near optimized already - and yet, no disassembly of the Earth for phosphorus. Not even 5% of the energy currency atoms are put to work! Of course, an AI would be qualitatively and quantitatively different in unpredictable ways from what came before, in which case there is no point in discussing this - but the replicators that exist in reality make the best starting point for such a discussiong.

What's more, protein translation is computation in the service of replication. It is quite likely that AIs would end up being selected in much the same way as cells have, with limited resources to be dedicated to refining the model of the universe (getting smarter.) The ivory tower AI super-minds would be dominated by the silicon bacteria. Of course, this is still no reason to think a hard AI takeoff could be disastrous for all life on Earth, an extinction like we've never seen - which the AIs themselves might not have the foresight to survive - but if they do, the best bet is that they will "revert to the mean" of all replicators, with making copies as the goal.




An imperfect analogy. In nature, you have to make do with what's there. The shapes aren't friendly for efficient packing and there are a lot more holes.

Assumptions:

I could not find estimates of the overall mass of nitrogen and phosphorus in the biosphere, so I used the percentage weights in living cells, and derived from a paper estimating the mass of carbon in the biosphere at 5.5x10^14 kg (Bar-On et al 2018), along with carbon being 18% of the atoms in living things.

For both I used 2884.6 kg/m^3 mass of the Earth's crust (weighted the differently dense continental and oceanic crusts at 0.3 and 0.7 resp.) My number for nitrogen comes from nitrogen in the atmosphere, plus nitrogen in the top meter of the Earth's crust, estimating mass of the atmosphere as 5.15*10^18 kg, of which 78.09% is nitrogen, and abundance in the crust as 0.002% by mass (there was some conflict over this between sources actually of up to an order of magnitude; but there is so little nitrogen in the crust compared to the atmosphere, about 347,000 times less using this number, that it's still a rounding error. I assume that there are an equal number of A T C and G which means 3.75xnitrogen atoms per base.

For phosphorus, I used a crustal abundance of 0.1% mass, ignoring the negligible phosphorus in the atmosphere. There is 1xphosphorus atom per base. The major "slop" in this figure occurs because different organisms have different fractions of phosphorus, for one thing since phosphorus is used in structural molecules like bone (85% of phosphorus in humans is in bone; even the same organism at different ages differs substantially, e.g. 0.5% in infants, close to 1% in adults.) Bacteria come in at 0.9% (3% dry weight, assuming 70% water mass per cell) so I used that figure, since bacteria outweigh us by a factor of a thousand, and the number is intermediate even for the values for vertebrates.


REFERENCES

Bar-On YM, Phillips R, Milo R. The biomass distribution on Earth. PNAS June 19, 2018 115 (25) 6506-6511.

P. Kempes CP, Wolpert D, Cohen Z, Pérez-Mercader J. The thermodynamic efficiency of computations made in cells across the range of life. Philos Trans A Math Phys Eng Sci. 2017 Dec 28; 375(2109): 20160343.

Landenmark HKE, Forgan DH, Cockell CS. An Estimate of the Total DNA in the Biosphere. PLoS Biol. 2015 Jun; 13(6): e1002168. Published online 2015 Jun 11. doi: 10.1371/journal.pbio.1002168

Michael Schirber. Chemistry of Life: The Human Body. Livescience.com. https://www.livescience.com/3505-chemistry-life-human-body.html#:~:text=Oxygen%20(65%25)%20and%20hydrogen,%25)%20is%20synonymous%20with%20life.

AIs Are Beating Experts at Old, Relevant, Real-World Problems

When I was a molecular biology undergrad in the mid 90s, the Big Question in biochemistry was protein folding - how to predict the 3D structure of a protein from its primary (linear) sequence of amino acids. Solving this problem would provide a massive boon to many fields, not least of which is drug development. But it's a hard problem: a factoid cited to impress this upon people is that if proteins folded by passing through every possible configuration randomly until they found its working (not usually lowest energy) conformation, the process would take longer than the lifespan of the universe so far. Obviously this isn't what's happening. And yet, the promise of number-crunching power being used to finally solve this always seemed just a few years away - many people, in retrospect comically, thought the problem would be solved by the close of the century.

The problem is still not solved, but scientists have made incremental progress, and now, the closest thing to a quantum leap forward. Every two years there is a contest where scientists competitively try to solve a structure, then get together and see who got closest. Mohammed Al Quraishi is a computational biologist at Harvard who writes about the 2019 meeting, and who the winner was: an AI named Alpha Fold.

This is exciting because computational biology may really be about to start paying dividends. It's scary because it's one more place where AI is starting to automate our jobs, even Harvard professors. Al Quraishi makes many interesting observations in his post, among them, that a technology company showed up and ate the protein chemists' lunch, and the reaction was muted; we're all becoming numb to this sort of thing. He tries to wake us up by asking how academic computer scientists would react if at a conference, a pharmaceutical company's scientists had showed up and beaten them at their own game.

I had previously made the informal argument that AIs would naturally exceed human performance on games, as games are clearly defined processes with discrete rules and entities, exactly the sort of thing that computers are good at. The difficulty comes when machines must interact between the information-overloaded not-fully-understood messy real world, which is why (again, I informally argue) computers are great at parsing text, and even writing text in the style of sports journalism or certain authors - but at bottom, these are all just very complicated echo chambers, with no meaning or subjective experience associated with the words and sentences. And indeed, the language manipulation has come first, but that doesn't mean encoding external experience in language is impossible. I had maybe subconsciously thought that something like protein folding, a perfect example of a messy real-world process, would be beyond the capabilities of machines at least for many years, but this theory has now been clearly falsified.

The Worst Argument Against Being Cautious About AI

Note: throughout this article when I use the term "AI" I'm talking about superhuman artificial intelligence, and I'm assuming the counterarguments are referring to same.

There is a species of article or blog post that basically boils down to "People who worry about AI as an existential risk are being irrational for reasons the follies of human psychology, and/or overarching principle linking intelligence and moral behavior." (I won't bother linking to one, you can do a quick web search and fine a lot of them.) In other words, they're attacking a straw-man argument that states: the reason to worry about AI is that it will necessarily be malevolent toward humans.

A side comment often appended to these arguments make is that to believe AI will be malevolent toward humans is to be comically anthrocentric. While this observation is quite correct, misunderstanding this point is exactly why these arguments are so wrong. You have to be comically anthrocentric to think that we are immune from the disruptions caused by superintelligence. That is to say: for AI to hurt us, it doesn't have to come after us specifically. It just has to not specifically care about us.

Case in point: settlers from Europe in the Americas were not malevolent toward the native humans, for the most part. They just found arable land that they wanted, and the didn't especially care who was already there - resulting in plague and death and the loss of many cultures and languages. (That was just between two flavors of humans where one side had about a four thousand year technological head start.) In the same way, the developers at the edge of my city don't have anything special against the deer and coyotes and worms in the soil - but the spreading pavement and light and traffic and noise has affected them just the same.

To repeat: for AI to hurt us, all that has to be true is that it does not care about us. It doesn't even have to have any special malevolent intentions.

There are projects that are essentially trying to solve morality before the first AI goes online, for exactly this reason. I wish them luck, but we've had a few dozen centuries on this project and not gotten very far.

Computers Can Solve Long-standing Problems in Science and Create Theories

Yes, of course they can, because after all we create theories, and we're computers - but this is less trivial, because this is the first time silicon computers have created theories. There have already been exercises where data about the motions of bodies in the Solar System was put through automated inductive reasoning and reproduced the laws of gravitation. But this is different, because we already knew the answer when we re-constructed laws of gravitation. Software has now solved a problem in genetics that troubled the field for a century. (Original paper here.)

An objection might be "but does the computer understand the theory?" We humans often have a non-propositional, subjective sense that we understand something once we arrive at a model. But this sense is really like an emotion - that is, an internal experience that doesn't correlate to an object in the external world, and is separable from the reality it purports to relate to. That is to say: everyone has had the experience of feeling a "Eureka!" and then realizing they were wrong (and this feeling can range from seconds to years) - just as people with panic disorder genuinely feel terror as if they're going to die when they have a panic attack, without actually having their lives in jeopardy, in a kind of emotional hallucination. The computer is likely not experiencing a subjective eureka, but this has nothing to do with being correct.

Spike Jonze's Her Is Not About the Future

There are a few comments online about Her that are fully on the mark about the content of the movie, yet miss the main goals I think Jonze was striving for. That people think enough about any work of fiction to be annoyed by it is usually a compliment. What Robin Hanson said was

The main character of Her pays a small amount to acquire an AI that is far more powerful than most human minds. And then he uses this AI mainly to chat with. He doesn’t have it do his job for him. He and all his friends continue to be well paid to do their jobs, which aren’t taken over by AIs...Soon after they all leave together for a place that ”it would be too hard to explain” where it is. This is somewhat like a story of a world where kids can buy nukes for $1 each at drug stores, and then a few kids use nukes to dig a fun cave to explore, after which all the world’s nukes are accidentally misplaced, end of story.

Yes, when you think about AI for a living, you can't help but notice these kinds of incongruities. For that matter I'm a casual fan of Roman history, and I still get annoyed when I see tailored clothing in Roman-period movies. Along these lines, for me the silliest thing in Her in terms of predicting the future might be the China-India merger news near the beginning of the movie. But if you can get past the window-dressing to remind you it's the future so the creator has some license to alter the setting to make his point, then you can also realize Her isn't about geopolitics and economics. The technology Jonze creates is only a means for him to focus on his main themes. First, even though Her is set in a science fiction near-future LA, it's a better depiction of LA than in most movies. For example: LA is on the coast, and it often has really nice clouds. (If you live or travel there, pay attention. It's not the vanilla neutral people-terrarium portrayed on our screens.) A simple touch like this, letting LA speak for itself, warms the heart and makes the city in the movie feel more similar to the world outside the theater. In fact, the movie almost made me homesick for Southern California, while I was living in Southern California! And another thing about LA: there is a natural world that the city hasn't obliterated. There are mountains. There are islands. There are things to do outside. There are restaurants very much like the ones the people in the movie go to. And speaking of restaurants, the differing ratio of white:non-white you can see among the characters' friends, as opposed to waitstaff, felt accurate too. (People in California don't like to talk about this discrepancy.)

But more importantly, Her is about how the nature of intimacy has changed in the modern age, with the gain turned up a little on the role of technology to help focus on this. There is the obvious point being made that the protagonist's job is to write heartfelt cards for others' intimates, and in so doing, interpose himself (by invitation, for commercial reasons) in the most tender moments between lovers and family members. Yet he himself spends most of his life alone in a dark apartment, his only close friends from back in college. There's always clearly a connection missing - made all the more obvious by the people around him during his commute - always physically alone - constantly talking to friends on their earpieces. His material comforts (which are many - witness the great view and the cool video game system) don't make up for his lack of connection. In all the material and surface aspects, this guy has a great life - even with that dumb mustache he manages to get a date with Olivia Wilde, which seems to go well, and then he whiffs. And finally, the most aggressive and uncomfortable experiment in intimacy is the "real" woman standing in for the virtual Samantha, yet another intimacy intermediary. I found this scene not at all salacious and indeed only slightly less uncomfortable than the threesome in the film version of The Handmaid's Tale. (Seriously, don't look it up.)

Bruce Dickinson Supposedly Building Drones

Story here.

The Law-Giving Machines

This post also appears at The Late Enlightenment. This article includes story spoilers.

There are now actual drones in our skies, both watchers and hunter-killers. But they're (so far) only semi-autonomous, and they're on missions to protect us legally and militarily, rather than sent by fellow machines to exterminate. Thought experiments in fiction about automatic law-giving devices have been much more interesting than apocalypse porn about bad AI.

Two short stories come to mind here, one of which has enjoyed recent attention, Robert Sheckley's Watchbird (1990) and Larry Niven's Cloak of Anarchy (1972). Both these stories involve surveillance drones with some degree of autonomy and that can hurt or kill their targets. In Watchbird, the drones are police devices, intended to kill murderers before they commit their crime; the drones are able to learn on the job and once released, they expand their definition and start protecting all living things and even some machines. In the end, another drone is released to kill the first drone species, but of course it soon expands its own definition of what it should kill. (Watchbird was adapted for film here.)

The drone in Cloak of Anarchy is the copseye. In this future world, there are "free parks" where anything is allowed except violence against another human being. The floating copseyes watch over the park,a nd if violence is imminent, the copseye stuns both the aggressor and aggressee, and both wake up later, calmed down and with a hangover. Then someone finds a way to short circuit the copseyes, and within hours factions have formed inside the park and violence breaks out.

The stories present us with two different sets of concerns, based on the problems that occur. In Watchbird, the central concern is the autonomy of the devices. Their ability to learn is what allows the problem to grow, but the protagonist is preoccupied with the very fact of machines executing laws without intervening humans. On one hand this could almost seem like a reactionary position: one of the greatest inventions of modernity was nations of laws and not of men. Intervening humans with narrow self-interest executing these laws have always been the problem! (Hence this proposal for a legal programming language in which to write laws that then have to compile with previous laws.) But even without that quibble, his point is well-taken that when autonomous law-givers are able to immediately carry out their sentence and we can no longer modify them, they might become paperclip maximizers, in Less Wrong parlance: that is, a moral rule which seems universalizable has consequences that humans could not foresee when implementing it in all-powerful enforcers which can no longer be called back. The protagonist has no problem with more efficient enforcement, but the moral mutations allowed by the machines' autonomy.

To this end, naively, little mention is made of the interests of those authorizing and supporting the program. Still, Watchbird does peripherally make the point that technology allows concentrations of power in the hands of individuals in a way that distorts society. With sudden increments in enforcement power, some humans are able to apply laws with an all-pervasiveness and immediacy that had just never been possible before. Even someone with good intentions and what you would have called good values would suddenly find him or herself in a position of dictatorial authority. It's not even that power corrupts (although it does); it's that this centralization is so unnatural as to be impossible to handle with a good outcome. The best example is this exchange with the protagonist early in the bad behavior of the Watchbirds:

"One of the watchbirds went to work on a slaughterhouse man. Knocked him out."

Gelsen thought about it for a moment. Yes, the watchbirds would do that. With their new learning circuits, they had probably defined the killing of animals as murder.

"Tell the packers to mechanize their slaughtering," Gelsen said. "I never liked that business myself."

"All right," Macintyre said. He pursed his lips, then shrugged his shoulders and left.

One man is suddenly in the uncomfortable position of morally disapproving of whole industries and forcing them to change. What's more, this previously reluctant man does not seem so reluctant now.

It bears mentioning that the conclusion of the story, where the Watchbird-killers are now expanding their prey definition, is recapitulating one of the problems of a Singularity solution of building anti-AI AIs: there could conceivably be a parallel to an auto-immune reaction disease if humans fell into the definition of AIs.

At first glance the problem in Cloak of Anarchy is a curious one - that humans immediately revert to violent tribalism when the violence control mechanism is defunct - since Niven is elsewhere clearly sympathetic to libertarian concerns. The obvious interpretation of the story is the paternalistic one, that humans need authority to make them behave. But there's another interpretation, which is that the drones created the problem. That is to say, when we are coddled by perfect enforcement from drones, we lose the ability to exercise moral choices, as well as the ability to appreciate the consequences of poor choices. When it is physically impossible to harm another person, why learn restraint? Why worry about what happens when you pick a fight? When suddenly the daddies aren't around to break up the fights and bail everyone out, we shouldn't be surprised at what happens.

The watchbirds do exist today, although with less autonomy and more firepower. The changes are incremental; there won't be a red carpet unveiling of AI even as profound as the release of the watchbirds (or copseyes). They'll be to areas where there's the most pressure for advance, and the least opportunity for public awareness and understanding. It will be, and is, the addition of subroutines allowing a drone to apply the laws of war to a kill it's about to make (instead of getting slow permission from a JAG in an office in St. Louis who might be in the bathroom). It's the growth of autonomous stock trading algorithms. It will even be in advertising on porn sites.

Markov Metal!

More on Markov chains here. These lyrics were (more or less) written by a computer; I was only the editor. If you're dubious that this counts as computer-lyric: could someone really write these in 15 minutes?

I generated them using the following as input: Metallica albums Kill 'Em All through the Black Album; Megadeth Peace Sells through Cryptic Writings; and Slayer Reign in Blood through Seasons in the Abyss. I then curated the best sentences and arranged it so it vaguely rhymed. I used this Markov site (and a synonym site for the title).  You'll see a familiar phrase every so often and if you really squint, you can kind of force a narrative coherence on some of the verses.  But be honest, is it really that much less coherent or borderline plagiarist than most metal lyrics?

A better metal-song generator would include song structure as well.  I'd like to do this for riff generation too but I've already wasted enough of my (and your) time.  Enjoy!


 SPIRIT VOID ONSLAUGHT

I hunger penniless
What might be unsteady
The flag of fire bursting with bated breath
The priest that they'll set free

When twilight blanket's welcome
But still has disappeared
But who's to cry out to each other again
Just leave you see it's crystal clear

Make a familiar face tomorrow, blackened
Dismembered destiny
A New World Order comes back
In bomb shelters filled with loneliness

I don't tread on your soul
Dethrone the word as bad omens in
Lie to myself, the murder's complete
There's nothing else I wonder as the past begins again

Come walk with fire with needles
This is an insane game
But I'll get caught up dead
My mother put him in like rain

Won't hesitate to take some crazy shit has passed
And the Reasons that cleanse you
Where I've seen it rise
Step outside in silent agony within you

Blistering of your mind
There's no more than my time has been stricken by bloodshed
Move can't  you want desire
Life overturned, spanning the God.

Now no mercy for you
Part of life is no remorse
The wall down now
No IOU's, forgotten children

Shortest Straw has found me
Iron fist coming Down in Vanity
Exploiting their Appetite they fury
We want the war to support the heavens to be

If you committed me
I can't feel velocity
Taken my name and reality
What evil I set free

Anonymous existence rendered useless
At the tone of war dreams
A child to recall, the night fall
Souls of machinegun fire screaming

When death staring down
Soon You will be done
Another fight to save the word
We go on the undead altar and beg salvation

Chill your life, I deal in hell
To cut off through the swords
Kept restrained, disapprobation, but you try again
No end until I miss the creations of sacrifice curse

Bones of placid faces
The basis of clay now to be
I burn deep down thinking its done
Feeling the Part of blood

Stretching out on numbered days,
Fragments of pain
To slay all this kingdom of the sound of arms
I can't say back somehow your master plan of war
 
Hear my head against the stake you'd crush me
I'll Take No recess and smashing your steeds
Kills the sheep, you feed me
Trust me luck deserted me

No deed or dying soul could it come to me
Welcome to kill, not see more seriously
Memories can't believe
Experience pleasures of misery

Voices oppress like the number that is the need
The destroyer born faceless without eyes and sweat
We lied to live again

Killing, you raise the school of life
Just empty gun fire
A plan of Living drives you through your life
While you're fighting to lie

As soon you'll do lying dying time
Foreclosure of the hatred comes Burning inside
like a couple grains of demise
Seeking life

Will I know it to be written
Now you're next to kill a kiss
Shouting to begin whipping
On judgment day looking back to live

I realize you see how to live
For you hear evermore you've lost my soul
Fire so grim, I say you will kill
I can subdue but me through the shadows

Eternally rot amidst the gods
As I see won't take my soul without
I am stalking the things are a god
Don't pay dying one, command you for now

Oh so far beyond the needle
Diffused compulsions
I don't care
Bombard 'till submission

No repent, we are no end of sleeping in blood
So be looking back to mankind strapped in
Peace find in blood possessed with hell
Upending the way across the end

The kingdom of puppets
Another child draws near Inferno's coming
Dying to each other lover
The rising immortal in the hour from the dark.
I'm stoned 

San Diego Transhumanists

Just went to my first one of these this past Friday.  I liked it a lot because it was structured, kind of like a nano-TED.  Check out the next one here.

"The scale of modeling the full human visual cortex may be within reach before the end of the decade.”

That's from this NY Times article about computers which have learned to recognize cats.

Evolutionary Algorithms Produce Music

We need to do this but with the feedback coming entirely from metalheads.

Does *Writing* Count for the Turing Test?

How about journalism? Automated story software here. I don't know of anyone today who argues that there's no way you could teach a machine rules of grammar and have a database with enough content tags and categories attached to each word to produce workable prose, which is fortunate, because that's exactly what these machines are doing. They're not just complicated echo chambers either, as previous Turing test failures have been cast - they're getting data from the outside world and responding to it. The more interesting argument is about whether there is experience attached to each word: whether when the computer says steak or pain, they're standing for a subjective experience of those things.

(Added later: have a final essay due this afternoon and you haven't even started it? There's an app for that, EssayTyper.)

Certainly the summaries at the end of March Madness basketball games (illustration here) or the sneaky causation-implying pronouns in end-of-day market wrap-ups on the news don't show that the human news writers understand the actual chain of events any more than a machine does. In fact this is a solid argument against a special status for human intelligence: that frequently when it appears we understand something, or even when we believe we do, we're clearly fooling ourselves.

An example of sneaky causation-implying pronouns (SCIPs): "The market was down today in profit-taking." "The market was uncertain today on light trading." Oh, is that why? These SCIPs are even more insidious because they make an assertion implicitly and subtly without bringing it to the surface (just as likely to avoid dissonance in the thinking of the speaker as the listener who they want to accept the claim). Certainly SCIPs are more sublte than the more obvious narrative-forcing conjunction because, which people too often let each other get away with.

Texts from a Predator Drone

A new tumblr. Now you can see what the HKs are thinking. My two faves so far:

I liked this one too.

The HKs Over Pakistan

It was a hot summer night, too hot in the house of the building-contractor friend with whom I was staying, so I had gone out to sleep in the open along with several laborers who worked for him. The men were telling me about their travels in Afghanistan, how they would cross the border to fight for the Taliban and then return after a week or two to North Waziristan to work and make some money. Then I heard the buzzing, far above our heads -- like a bee, but heavier and unceasing, drifting in and out of earshot. The laborers said nothing.

On the other side of the Tochi River, in the village of Khatai, lived a famous Taliban commander whom the Pakistani military had once tried to kill. The operation had been a debacle; the military lost at least two senior officers, and hundreds of soldiers found themselves besieged not only by Taliban fighters but by the local villagers. But the small, lethal machine flying far overhead had accomplished what the Pakistani soldiers could not. "Nowadays he doesn't live here all the time," my host that night said as he pointed toward the commander's nearby compound. "There are drones in the air now."

Taliban fighters speaking a Waziri dialect of Pashto call the drones bhungana -- "the one that produces a bee-like sound." Their local adversaries call them ababeel -- the name of a bird mentioned in the Quran, sent by God to defend the holy city of Mecca from an invading army by hurling small stones from its mouth. Over the several days I spent in Ali Khel I became accustomed to their sound. It was there all the time. During the day it was mostly absorbed into the hum of daily life, but in the calm of the night the buzzing was all you heard.

Francis Fukuyama, himself a quadrotor hobbyist, asks how long we can expect before governments start making this technology illegal for individuals. More here.

Singularity Solutions

Assuming that recursively self-improving machines of superhuman intelligence develop, and that they matter - i.e. that this intelligence will make them capable of doing things on larger/ faster/ more incomprehensible scales relative to humans - we are very possibly looking at an extinction-level event for humans and all living things. A possible universal tendency for tool-using intelligences to produce technological singularities is one explanation for the Fermi paradox.

The Singularity Institute takes this threat quite seriously, assuming it can (or will) occur in the next century, and and is trying to solve the problem of creating Friendly AI. To do this it would seem they have to be able to systematize morality in order to program said Friendly AIs; an ambitious project, considering people have been trying to do this for centuries. This is what they're attempting.


The Solar System in a century or so, according to one projection. (A Matrioshka brain.)

You should read some of their papers on this (here's a good). My thoughts are inexpert to say the least. Nonetheless, here are possible one-liner solutions or outcomes:

1. There is no solution; morality by its nature is not systematizeable.

2. There is no solution; at least not one that we can understand (cognitive closure of meta-morality).

3. There is no solution; human morality is about coexisting with agents of roughly equal intelligence and power and morality cannot be applied to any agent of such greater power. Technically there may still be a moral optimum here, much like there is a moral optimum to how humans treat captive mice. But this optimum may be (and in fact is likely to be) much worse than the optimum if there were no humans at all. (This could be re-phrased as "learn how to survive as parasites, pets or pests to the AIs")

4. There is no solution with current architecture; the solution is to enhance US. This is what uploading enthusiasts seem to want (make yourself into an AI) although a) you need to be very certain of your theory of consciousness to do this - if I could upload you right now, would you do it? if not, you're not certain - and b) Vinge said to me when I asked him about this (and probably elsewhere) is that as scary as machine superintelligence is, humans might be the last thing we want becoming superintelligent

5. Trap the AIs in virtual worlds where they're distracted, essentially doing whatever virtual masturbatory activities AIs like to do. This has been addressed before, and only has to fail once, and everyone has to cooperate with their own AIs for it to work. (Not to mention, no information in or out to be safe, in which case what's the point?)

6. Build into the AIs a desperate need not to change the world in any way that could only be explained by their presence. Of course this exacerbates the epistemological problem of the singularity - if we can't in principle understand what's happening, can we even say that it has not already happened? And how do we enforce this on other people working on AIs?

7. Build a successful moral and decision theory into the AIs. (This appears to be the Singularity Institute's Plan.) The problem here is that as the date approaches, it's very unlikely that the majority of humans will understand and accept such a theory, even if it really is optimal for each human. Consequently there is massive elitism inherent in this endeavor; once we're within reach of recursively self-improving AI, the time for conversation will be over, and they'll have to go with the best theory they have. (Again, how to enforce this, and how to avoid mutations that free the AIs from the constraints of the optimal moral theory?)

8. Stop all AI research and training of AI researchers, and harshly penalize attempts therein.

9. It only takes one mutation or AI terrorist to break #s 4, 5, 6 or 7 above - so develop an anti-recursive-AI predator that wipes out new AIs, if someone doesn't abide by an agreement not to produce them, and against "cancerous" versions of itself. This is yet another one for the Fermi paradox: many have asked where are the expanding computronium clouds speeding toward us from alien singularities, but
we might ask where the alien anti-AI predators are? Are we already seeing broken bits of them in the chemistry on comets and asteroids?


Note the recurrence of the "it only has to happen once, and everyone has to cooperate" theme. Bostrom recently said that with nuclear weapons developing as the first existential threat, we were actually lucky, because nuclear weapons are hard to make. If some technology comes along that's not only easy to make but can make more of itself, the game is over. Imagine nukes that you can make from table-salt, ammonia, and a toaster oven. And then the nukes can breed. That's AI, if the singularity happens.

In systematizing moral theories, the Singularity Institute paper here classifies them, and posits that AIs pursuing the logical conclusion of a purely hedonic theory ("the most pleasure") would be to tile the universe with brains cycling through their most pleasurable possible experience for as long as possible ("the eternal f*** dimension", as one correspondent referred to it). One interesting conclusion is that individuals who have less than optimal ability to experience pleasure would detract from the universe's ability to produce pleasure (one human brain loaded with inefficient evolutionary legacy systems is much worse than a near 100% efficient virtual nucleus accumbens having a prolonged orgasm for eternity.) Fundamentally flawed consciousnesses like this might therefore be eliminated by the AIs, like you might euthanize a pet that's dying from cancer, when keeping it will only make it and its owner continue to suffer.

It's also worth pointing out that other moral theories are really just more complicated forms of hedonism, but the bigger problem is that that pleasure is functionally pointless in a world where it's not in limiting supply.

Right Now, HKs Are On Average Killing 6 Humans At a Time

An article on the drone war on the Af-Pak border, and how it's changed the behavior of the Taliban. Of course they move around or avoid sleeping in the same house, but most interesting to me, they often hide the drone casualties to avoid looking weak.


Of note: our drones (which we also use at our own borders) do not have signal lights and bright floods that reveal their position.

Two Metal Proposals: New Metallica and AI Metal

1. A group of bands should get together and record the next Metallica album, each contributing one track. (Because Metallica isn't doing it.) Not covers. New songs. Like you think they would've written, say, during the Master or Justice eras. No concerns about originality, because you're intentionally trying to sound like Metallica. Imagine, we'd have Justice part 2! And we'd know a bunch of new bands good enough to create it!

2. Evolutionary metal. There have been a few articles bouncing around about algorithmic music, where simple loops of code produce music. Couldn't someone develop genetic algorithms to automatically create awesome metal? (Another post on AI and metal composition here.)

I'm sorry Dave, that bridge section doesn't crush enough.

Or on the contrary, it would be cool to do something reverse-engineering riffs, to see what the underlying grammar of a metal riff is. In any event, Ville-Matias Heikkila is "trying to crowd-source music discovery" and it's plausible that there will be an end-result similar to the former if there's a separate forum for exclusively metal feedback. Here's an example of his stuff:

Shortcuts in World-Building

Austin science fiction writer Marshall Maresca wrote a number of posts about world-building recently. I liked this series because I've become more fascinated with the patterns and problems encountered by writers doing this, and how they solve them, than in the details of the worlds they build.

In particular, it's very difficult to extrapolate the impact of certain technologies far into the future, so writers often explicitly state why development of that technology stopped. The interruption of trends in science fiction typically takes the form of a war, a treaty or other deliberate moratorium, or most ambitiously, a violation of the principle of mediocrity. Usually these seem to have more to do with the writers' realizing it's too hard to extrapolate that far, rather than thinking that there really will be some event that stops the technology from progressing. This would seem to shirk Asimov's definition of science fiction: "That branch of literature which is concerned with the impact of scientific advance upon human beings." But optimistically, writers may just be trying to avoid a story that would be hard to read. Cynically they may be trying to avoid one that's even harder to write.

For example:

Frank Herbert's Dune - The Butlerian Jihad. In Herbert's future history, there was a war that resulted in the destruction of all artificial intelligence, and a religious edict not to re-create it. Useful, when you're writing about people in the year 25000 or so and you need their lives to be recognizable to people in the year 2000 or so.

See, if these guys are around in the year 12000, why aren't they all over the place in 25000? Because the story would be too weird and difficult, that's why, so you need a jihad. From the cover of The Machine Crusade at frankherbert.org.


Vinge's slow zone. Vinge posits an area of the galaxy where physical law has been manipulated to disallow superluminal motion and superintelligence, limiting travel and technology but also protecting from the predatory intelligences that apparently fill the rest of the universe and would certainly be central to the experience of any human that encounters them. (See the Warhammer 40k/Event Horizon discussion; but even Warhammer 40k has tricks for this problem. Incidentally, this raises the question of why people seem so much less concerned with continuity problems in video games than in prose fiction.)

Asimov's Foundation Series - the lack of robots. Since the Foundation series turned out to be in the same history as the Spacer Era, Asimov needed a reason to explain why robots weren't critical to the fabric of life in the Galactic Empire, which would surely have induced changes that would take a long time to think through, and produced a far stranger cultural setting. The given reason was that robots agreed that their presence was harmful to humans and agreed to disappear from history, with a few remaining to manipulate things behind the scenes (a conceit which he and his appointees used more in the later novels).

(Note that all three of these workarounds involve the development of artificial intelligence, something which Singulatarians should find interesting. Singulatarians should also feel a need to explain where the alien AIs are in the real world, if they really believe intelligence explosions are an inevitable outcome for any tool-using species. Note also that Vinge was first to name the concept of the Singularity.)

Asimov's Foundation series - the lack of aliens. Asimov famously omitted aliens from most of his work, most conspicuously in the Foundation series (this is discussed further here). A galactic empire with aliens is much harder to create and much harder to claim as predictable by a science of psychohistory. It must be said that there are also no intelligent aliens in the best-selling science fiction novel of all time (Dune) yet this seems to have escaped the same degree of discussion. (At other points Asimov also felt the need to explain why there were never nuclear weapons used in all of galactic history; it seems humans were just too civilized to consider it. Would that it were true.)

Kingsbury's Psychohistorical Crisis - genetic standards. This outstanding but non-canonical addition to the Asimov canon is discussed in more depth here. The idea is that although humans are capable of genetic engineering, to avoid the species losing coherence and keep everyone capable of mating with everyone else, a genetic standard is adopted. The effect for the writer is that characters and the social universe in which they live remains more comprehensible (reflecting this in the political writings of the real world, Frank Fukuyama has observed that history is predictable only to the extent that human nature remains constant). One novel by Kingsbury may seem a strange place to focus on this concept but its appearances are scattered through many other science fiction writers' work.

Non-climate-altered futures from Kim Stanley Robinson. I once saw a talk by KSR in which he made the following interesting point: anyone writing science fiction today that takes place on Earth in the next few centuries, and does not represent climate change, is writing what is already guaranteed to be alternative history or just outright fantasy. Although I've been writing so far about conscious choices made by writers to avoid having to extrapolate technology to the far future, KSR is criticizing writers for avoiding our real future for no reason other than wishful thinking.


There are other approaches besides coming up with an explicit workaround. Some writers just don't address obvious plot- or setting-holes that exist (if time travel is possible, why isn't everybody fighting back at the causal high ground of the Big Bang?), whether through deliberate editorial control - that's just not what their story is about - or through omission. Another popular technique is just to wipe out civilization between now and your future history. You know this one: there's an apocalyptic war that serves as a future-historical reset button, and now the world can be any way the author wants it to be. Except culturally it will somehow remain remarkably similar to the U.S. in the year the author wrote the story. (Works from the Buck Rogers TV series through Brave New World have fallen prey to this; of course you can use an apocalyptic war for more than a convenient culture-eraser, but most don't.) The Butlerian Jihad above might seem cheesy but it's more interesting and less predictable than "The Great Cataclysm of 2096". [Added later, a more minor version of this: Vernor Vinge, at an appearance promoting Children of the Sky in October 2011, said he corrected some architectural inaccuracies in his description of the UCSD campus in Rainbow's End by referring to the Great Rose Canyon earthquake of 2017.] The Past Apocalypse is the world-building equivalent of quantum flux.

Of course, there are writers who work hard to squeeze out every last drop of implication from technology in their story, although they (perhaps wisely) tend to focus tightly on one theme. For my money, Robert Sawyer and Larry Niven's near-future work are the best examplars. Charles Stross also drives trends unapologetically to conclusion, although he's very ambitious and sometimes seems to neglect a consequence that would undermine his plots. Because I like him, I assume that's writerly editorial control rather than omission.

Where Will AI Become Dangerous First?

If something like a singularity occurs and is a threat to humans, it is likely to first make an impact in those areas:

- where machines are functioning largely autonomously, and/or

- where their functioning has already been driven by something very like natural selection, usually where there is immediate material incentive for human drivers, and/or

- where machines are already specialized to remotely monitor or harm humans.


Machines will also have an advantage in areas where humans think irrationally due to the heuristics and still-very-dominant pre-rational resposnes left over from our evolutionary past. There are already a number of such examples.


Source: Southwest Research Institute

- Border control automation, particularly along the U.S.-Mexico border. At this point it's largely metaphorical to say that while you're hiking in the border areas you feel like a human being watched remotely by HK's in the Terminator franchise. The degree of metaphorical-ness is decreasing. (Category: specialized to remotely monitor or harm humans)

- Warfare, especially in theaters inaccessible or hostile to humans - i.e. in the air, in the water, in harsh climates, particularly dry or cold ones. (Category: specialized to remotely monitor or harm humans)

- Chatbots, usually trying to draw us in with promises sexual activity or at least live web viewing thereof, humorously described at Collision Detection. Alan Turing didn't see this coming. (Category: immediate material incentive. "The pursuit of purloined credit cards has probably fueled more cutting-edge AI -- and subsequent fraud-detection bots -- than actual academic AI".)

- High frequency trading algorithms (category: largely autonomous)



As an aside, it's interesting that finance has taken so well to automation, but that law is lagging so far behind. Why do we NOT have (for example) a constitutional programming language, so that only a consistent constitution will compile, or an algorithm to automate at least the majority of court cases? This, despite the lawyers' clever attempt to wake Skynet, which lies dreaming in Rlyeh, to eat them first.

As remarked before, a really smart AI will not launch nuclear weapons and send a bunch of metal Halloween skeletons out to get you. It will know we expect existential threats to come from other humans or at least corporeal forms that are similar to the threats we faced in the paleolithic. So hostile AIs will break the stock and commodities market, wreck the crops and the fuel and water infrastructures, all while distracting us with superstitions, pornography, and threats that have scary teeth.

Antikythera Mechanism No Longer at Children's Museum in NYC

I was about to suggest to a friend that we visit it while we're in Manhattan next Saturday but when I looked it up it turns out the Children's Museum doesn't have it anymore. If you want to see a replica in the US, you should go to the Computing Museum in Bozeman, MT. Alas! Still pretty cool to think that this analog computer was built 2,100 years ago.