The Worst Argument Against Being Cautious About AI

Note: throughout this article when I use the term "AI" I'm talking about superhuman artificial intelligence, and I'm assuming the counterarguments are referring to same.

There is a species of article or blog post that basically boils down to "People who worry about AI as an existential risk are being irrational for reasons the follies of human psychology, and/or overarching principle linking intelligence and moral behavior." (I won't bother linking to one, you can do a quick web search and fine a lot of them.) In other words, they're attacking a straw-man argument that states: the reason to worry about AI is that it will necessarily be malevolent toward humans.

A side comment often appended to these arguments make is that to believe AI will be malevolent toward humans is to be comically anthrocentric. While this observation is quite correct, misunderstanding this point is exactly why these arguments are so wrong. You have to be comically anthrocentric to think that we are immune from the disruptions caused by superintelligence. That is to say: for AI to hurt us, it doesn't have to come after us specifically. It just has to not specifically care about us.

Case in point: settlers from Europe in the Americas were not malevolent toward the native humans, for the most part. They just found arable land that they wanted, and the didn't especially care who was already there - resulting in plague and death and the loss of many cultures and languages. (That was just between two flavors of humans where one side had about a four thousand year technological head start.) In the same way, the developers at the edge of my city don't have anything special against the deer and coyotes and worms in the soil - but the spreading pavement and light and traffic and noise has affected them just the same.

To repeat: for AI to hurt us, all that has to be true is that it does not care about us. It doesn't even have to have any special malevolent intentions.

There are projects that are essentially trying to solve morality before the first AI goes online, for exactly this reason. I wish them luck, but we've had a few dozen centuries on this project and not gotten very far.