The Incentive Problem

Here is the part that should make us most uncomfortable: none of what has been described in the preceding chapters requires evil people.

The AI race does not need villains. The erosion of privacy does not require a conspiracy. The automation of warfare does not demand a mad general. Every dynamic described in this book can emerge, and is emerging, from actors who believe themselves to be behaving responsibly.

Consider the position of a CEO running a major technology company. Her researchers tell her that a transformational AI system is within reach. She also knows that a competitor is pursuing the same breakthrough. If she slows down to conduct more safety research, the competitor may reach the milestone first. If the competitor succeeds, they will gain enormous economic advantages, advantages that may be irreversible in a winner-take-most market. If she believes the technology will be built regardless of what her company does, she may reasonably conclude that the safest course of action is for her organization, with its safety culture, its alignment research team, its institutional commitment to responsible development, to build it first.

This is not villainy. It is competitive logic. And competitive logic can produce collectively dangerous outcomes even when every individual participant is acting rationally within their own frame of reference.

The same dynamic operates between nations. If one country believes its rival is developing AI-enabled military capabilities, choosing not to develop its own may feel like strategic negligence rather than principled restraint. Each side tells itself that its own development program is defensive, that it is pursuing capability in order to deter, that falling behind would be the truly dangerous outcome.

And so everyone keeps moving forward. Everyone talks about safety. Everyone insists on the importance of control. Everyone warns that the other side must not gain an unacceptable advantage. And the pace of development continues to accelerate.

This is the trap, and it operates not at the individual level but at the structural level. Each participant behaves rationally, and the collective outcome becomes irrational. The logic is identical to the one that drove nuclear proliferation, that continues to drive climate change, that defines every tragedy of the commons: the incentives facing individual actors diverge from the interests of the group.

Extreme wealth amplifies this dynamic in a specific and underappreciated way.

A person with ordinary resources who becomes convinced of a grand vision, whether colonizing Mars, building artificial general intelligence, or restructuring the global financial system, can do relatively little about it. Their ambition is bounded by their means. But a person with ten or fifty or two hundred billion dollars faces no such constraint. If they become convinced that humanity must reach Mars, they can finance the rocket company. If they believe AI must be developed at maximum speed, they can build the lab. If they decide that existing governments are too slow, too cautious, or too corrupt to manage the transition, they can attempt to build around them entirely.

The capacity to act on a vision becomes proportional to wealth. But there is no corresponding mechanism, no natural law, no market feedback, no institutional check, that makes a person's judgment proportional to their power. A billionaire is not ten thousand times wiser than a person with a hundred thousand dollars. They are simply ten thousand times more capable of acting on whatever wisdom or folly they possess.

That asymmetry is the engine of the problem.