Faster Code, Shakier Delivery: Reading the 2024 DORA Report

The 2024 Accelerate State of DevOps report landed on October 22, and its AI findings deserve more attention than the headline adoption numbers. More than 75% of respondents rely on AI for at least one daily professional responsibility. More than a third report moderate to extreme productivity gains.

And yet, as AI adoption increased, the report estimates delivery throughput fell by 1.5% and delivery stability fell by 7.2%.

A week later, Sundar Pichai told Alphabet's investors that more than a quarter of all new code at Google is now generated by AI, then reviewed and accepted by engineers. AI-written code is not a pilot any more. The DORA data is the first large-scale look at what it does to delivery, and it is worth reading carefully.

The good news is real

DORA estimates that a 25% increase in AI adoption is associated with a 7.5% improvement in documentation quality, a 3.4% improvement in code quality and a 3.1% improvement in code review speed. Those are meaningful gains in exactly the places teams have always under-invested.

So this is not a story about AI making code worse. On the measures closest to the developer, things got better.

So why does delivery get worse?

The report's own explanation is the most useful line in it: improving the development process does not automatically improve software delivery, at least not without proper adherence to the basics, like small batch sizes and robust testing mechanisms.

Read that with an operator's eye and the mechanism is familiar. If an assistant makes it easy to produce more code per change, changes get bigger. Bigger changes are harder to review, harder to test, riskier to deploy and slower to roll back. Throughput drops because larger batches move more slowly through the pipeline. Stability drops because when a large batch fails, more breaks at once.

Trust is part of the picture too. 39% of respondents reported little to no trust in AI-generated code. Low trust plus large diffs is a recipe for either slow reviews or rubber stamps, and neither helps delivery.

The tooling is not standing still, either. At the end of October, GitHub announced that Copilot users can choose between models from Anthropic, Google and OpenAI, with organizations able to control which models their teams use. More choice is welcome, but it also means more variation in how changes are produced. That makes a consistent delivery process more important, not less.

Getting the basics back

None of the fixes are new. That is the point. AI raises the cost of skipping them:

• Cap batch size. Set a soft limit on lines changed per pull request and ask for large generated diffs to be split into reviewable steps.
• Merge small and often. Trunk-based development with short-lived branches keeps AI-produced work from piling up into one big release.
• Make tests the gate, not the reviewer. Fast, reliable automated tests on every change, with flaky tests treated as defects.
• Shorten the feedback loop. If CI takes 40 minutes, people batch changes to avoid waiting. Fast pipelines make small batches the easy path.
• Decouple deploy from release. Feature flags and progressive rollouts let small changes reach production safely before they reach everyone.
• Track the four keys alongside AI usage. Deployment frequency, lead time, change failure rate and recovery time, broken out by team, next to adoption data.

Platforms are the lever, with a caveat

DORA's platform engineering findings point the same way. Internal developer platforms increase developer productivity, and they work best when built with a product mindset: user-centred design and developer independence. The caveat is that adopting a platform can bring a temporary performance dip before gains appear as it matures.

That is worth planning for. A platform that bakes in small batches, fast tests and safe rollouts is how you make the basics the default rather than a discipline every team has to rediscover. Just do not expect the numbers to improve in the first quarter.

What we recommend

Before you expand AI tooling further, measure where you stand. Pull your four key metrics for the last year, chart pull request size over the same period, and check whether the two moved together. If changes have grown, start with batch size and pipeline speed. They are cheaper than any new tool and they address what the data says is going wrong.

Then treat your delivery platform as the place where AI gains are either kept or lost.

If you would like a second opinion on your delivery metrics or your platform roadmap, the HAZL Solutions team is always glad to compare notes.