AI coding tools let people build software much faster, and they skip the step that used to make someone understand the code. Fixing, securing and documenting software all relied on that understanding. The time saved at the start is likely to be paid back later, often by people who had no say in it.
London taxi drivers and GPS
London black-cab drivers spend years learning what the city calls the Knowledge: roughly 25,000 streets within six miles of Charing Cross and how they connect. In 2000, Eleanor Maguire's team scanned licensed drivers' brains and found that the rear of the hippocampus, the part that holds spatial maps, was larger than in people who didn't drive cabs, and grew with years on the job.
GPS removes the study. People with more lifetime GPS use have worse spatial memory when they find their way without it, and in a small group retested three years later, those who used it most had declined most. They still reach every address, so they don't notice the loss until GPS stops working.
Writing code used to build understanding in the same way. To get a program running, you had to know what each part of it did, so understanding came with the work and nobody had to plan for it. Code review, debugging and most of the security in small teams have relied on that understanding ever since. AI tools now do most of the writing, so the understanding no longer comes with it.
Where the speed-up was measured
In 2022, GitHub ran a controlled experiment in which developers asked to build a small web server in JavaScript finished 55.8% faster with Copilot than without it. The figure is often quoted as proof that software now costs half as much to make. The task had a clear spec, no existing code, no users and no history, so it measured only the first few hours of a program's life.
Most of a program's cost comes later. Robert Glass put maintenance at 40 to 80 percent of the lifetime cost of software, most of it spent adding features to code that already runs. That work depends more than any other on someone knowing why the code was written the way it was. The tools make the cheapest phase faster and do nothing for the later phases that cost more.
The extra cost is starting to show. An analysis of 153 million changed lines found that the share of code rewritten or reverted within two weeks rose after AI assistants arrived, and was on course to double its 2021 level in 2024. Code undone that quickly has to be paid for twice.
What this means for security
In most small teams, security depends on what the person who wrote the code knows about how it handles a password, a file upload or a search box. They used to learn that by writing it.
When Stanford researchers gave people an AI assistant for a set of security-related coding tasks, the assisted group wrote less secure code on most tasks and was more likely to rate its insecure answers as secure. The overconfidence matters more, because people can't see what they don't understand. A developer who writes a login form by hand has to decide how it stores passwords, so they know a decision was made, even if it was a poor one. A developer who generates the same form with a prompt never sees that decision and has no reason to check it.
What documentation used to show
Before Tenable bought Bit Discovery in 2022, we at HOBI worked with Moonlight Labs on the platform's documentation: feature guides, data sheets and case studies that explained each new feature to the people selling it and the people buying it. Each document came from a chain of explanation. Engineers explained the change to us, we explained it to sales, and sales explained it to buyers.
That chain was slow and it cost money, and each finished document showed that at least two people understood the change well enough to explain it plainly to a third.
A model can now write the same kind of document, or a release note, straight from the code changes, and it will often be accurate. It no longer shows that anyone understood the change, because nobody had to. The company keeps the document and loses that evidence.
Who carries the risk
In 2001, the Cambridge security engineer Ross Anderson argued that systems end up insecure when the people who could protect them aren't the people who lose when they fail. His clearest example was ATMs. American banks had to prove that a customer disputing a withdrawal was mistaken or lying, which gave them a reason to protect their systems. In Britain the customer had to prove it, the banks grew careless, and fraud followed.
AI-written software has the same problem. The person who decides whether a generated app gets read before launch is the person who saved the time. The people whose emails and passwords sit in its database had no say, and they carry the loss if it fails. The software is cheap for the person who builds it, and the risk falls on the people who use it.
What teams can do about it
London kept the Knowledge. Every phone in the city gives turn-by-turn directions, and cab drivers still spend years learning the streets.
Every team that uses AI to write software faces a similar choice, often without noticing. It can spend all the saved time on more features, or spend some of it on the understanding that used to come for free. In practice, that means someone who can explain each change in writing to a person who didn't make it, before it ships. That explanation was the slowest part of the old way of working, and it did more to keep software safe than anyone had accounted for.