AI Safety or Regulatory Capture?

I’d like to share a few observations about the reported dangers of Artificial Intelligence and the solutions proposed to address them.

The people who are developing the world’s most powerful AI models are working to make them smarter and more capable, while some of those same people know the technology could become extremely dangerous.

Jacob Coxon, a former employee of both OpenAI and Anthropic, resigned from Anthropic last week, warning that frontier AI companies are developing models that could one day be used to destroy humanity.

Dario Amodei, CEO of Anthropic, has also raised the alarm and has called for stronger safety measures, for advanced AI systems to be tested outside of the laboratory, and for more cooperation between governments. Sam Altman from OpenAI has issued similar warnings about where the technology might be going.

The Risks Are Real
Artificial intelligence presents real risks to humanity. Cybersecurity is one example.

A powerful AI model could help hackers find weaknesses in computer networks, develop new attacks, and automate work that previously required a team of skilled people. Imagine giving a hacker an assistant that never sleeps, can read millions of lines of computer code, and can test thousands of possible attacks in a short period of time. That’s a legitimate concern. AI won’t suddenly crack every encrypted message on Earth, but it is getting better at identifying vulnerabilities and finding ways around digital defenses.

The same problem applies to biological and chemical weapons. AI is becoming more knowledgeable about biology, chemistry, and genetics. That knowledge could help researchers develop drugs, diagnose diseases, and discover new treatments. But a terrorist organization, hostile government, or criminal group with access to powerful AI could do things that were previously unthinkable.

When Amodei and Altman warn that AI poses a danger to humanity, they’re not making things up. There are reasons to be cautious.

But that’s not the whole story.

The World’s Strangest Sales Pitch
OpenAI and Anthropic have both been positioning themselves to become publicly traded companies. When a company goes public, it must convince investors its product is better than the competition. For an AI company, that means proving its model is smarter, more powerful, and more advanced than everyone else’s.

That typically sounds like this: our product is faster, more useful, less expensive, or easier to build and maintain than our competitor.

The AI industry has a strange sales pitch: Our model is so powerful it might destroy humanity.

Imagine a car company telling investors: “Our new model is so advanced, we’re not sure anyone will be able to control it.” Or an aircraft manufacturer saying: “Our new jet is so powerful that we’re not sure it can be safely flown.”

Crazy, right?

Yet with AI, technological superiority can be expressed as how dangerous it can be if used improperly.

Who Writes the Rules?
Altman, Amodei, and others insist formal oversight of advanced AI is needed. That could include “independent” testing, government safety standards, restrictions on the most powerful systems, and eventually some form of international regulatory body.

Some of that may be needed, but here’s the problem:  Large companies usually coexist well with regulation. Small companies generally do not.

Let us assume that the government establishes a complicated set of policies and procedures for advanced AI. A company might need to spend millions of dollars on testing, legal compliance, outside audits, security measures, and government certification before releasing a new model.

OpenAI and Anthropic can afford that, but what about a startup?

A small company is unlikely to have hundreds of millions of dollars to spend satisfying regulators. Regulation that was ostensibly created to control the biggest companies often ends up protecting those companies from competition.

There’s another problem. If governments create a new agency to oversee artificial intelligence, they’ll need experts. Where will the experts come from? Who will write the regulations, design the testing, and certify compliance?

Most of the people who are qualified for those duties work at places like OpenAI, Anthropic, Google, and Microsoft. Many of the regulators will come from the very companies they will be tasked with regulating.

There’s a name for this problem: Regulatory capture.

David Sacks has been warning about it for years. He argues that if the largest AI companies convince the government that their technology is so dangerous that only heavily regulated companies should be allowed to develop it, they could effectively build a wall around themselves. Once the regulatory wall is built, it locks out new competitors.

Liability
Another part of the debate concerns product liability. Let’s imagine an AI company releases a powerful model. The model is later used in a massive cyberattack, or causes some other kind of catastrophe. Who is responsible?

Right now, the company that built and released the model is on the hook if its negligence contributed to the disaster.

But if a government agency creates regulations, oversees testing, and certifies compliance, the company can argue that when something terrible happens, it followed the rules and isn’t responsible for the damages.

For an AI company facing potentially enormous lawsuits, there are billions of reasons to shift liability to someone else.

International Regulation
Some of the proposed solutions go beyond the United States. If advanced AI really can threaten humanity, international regulation would be the logical next step. But that creates another set of problems. America could slow down AI development while China keeps going. Europe could impose penalties on any nation that ignores its regulations.

An international agency might restrict who can build advanced AI models. It might control the sale of advanced computer chips and determine where data centers can be built.

Do we want to give that kind of power to an unelected international bureaucracy?

The debate is sometimes presented as a binary choice. Either AI is a dire threat to human existence and needs immediate regulation, or the entire safety discussion is nothing more than fearmongering by big tech companies.

Two things can be true at the same time.

Artificial intelligence can present legitimate dangers. At the same time, some of the people warning us about those dangers can have financial and political incentives of their own.

Bad actors using powerful AI systems for hacking, cyberwarfare, or biological threats are a real concern, but regulatory capture is equally concerning.

The right response is neither panic nor complacency. It’s scrutiny and discernment. We live in an age when governments, corporations, political movements, and advocacy groups are influencing public opinion. Fear is an effective tool for doing it. Before making an emotional commitment to a cause, understand who is promoting it, what they want you to do, and who benefits if they succeed.

If an AI company believes its latest model is too dangerous to release, it can choose not to release it. If certain capabilities present specific dangers, governments can address those dangers directly. (That becomes a question of whether you trust your elected leaders.) If companies behave irresponsibly and cause harm, they should be held accountable.

We should be cautious about letting apocalyptic predictions become a blank check for a regulatory system controlled by a small group of governments, corporations, and experts.

Artificial intelligence may become the most powerful technology mankind has ever invented. The institutions built to control it could become even more powerful because they would ultimately use the tools they create.

We should examine both of them with equal care.