When AI Writes the Software, Who Verifies It?

Andrej Karpathy, one of the world's most respected AI researchers, admits he no longer reads the code AI writes for him. "I 'Accept All' always, I don't read the diffs anymore." That confession captures something unsettling happening across the software industry.

Google, Microsoft, and Amazon are already generating 25 to 30 percent of their new code using AI. Microsoft's CTO predicts that figure will hit 95 percent by 2030. But nearly half of AI-generated code fails basic security tests, and the humans who used to catch errors are increasingly stepping aside.

The stakes are high. One bug in OpenSSL, known as Heartbleed, cost the industry hundreds of millions of dollars. Poor software quality already costs the US economy $2.41 trillion annually, and that figure was calculated before AI started writing a significant share of new code.

A growing movement of researchers argues the answer is mathematical proof, not just testing. An AI recently converted zlib, a compression library embedded in countless systems worldwide, into formally verified code with minimal human guidance. The proof guarantees correctness across every possible input.

The question facing the software world is no longer whether AI will write its code. It is whether anyone will be able to prove that code actually works.