What the CrowdStrike Outage Teaches Us About Shipping AI-Generated Changes

At 04:09 UTC on July 19, CrowdStrike pushed a content update for its Falcon sensor on Windows. It was reverted at 05:27 UTC. In those 78 minutes, Microsoft estimates that roughly 8.5 million Windows devices were affected, less than one percent of all Windows machines, and the disruption that followed was felt around the world.

It is tempting to file this under "security vendor bug" and move on. That would miss the part that applies to every engineering team, and especially to teams that are starting to let AI write their changes.

The defect got through testing. What turned a defect into a global outage was that the change went everywhere at once.

It was configuration, and it shipped like configuration

The faulty update was Channel File 291, delivered as what CrowdStrike calls Rapid Response Content. CrowdStrike's own guidance describes it as configuration data, not code or a kernel driver. That distinction is exactly the problem. In most organizations, code goes through staged rollouts and change review, while configuration, feature flags and content updates take a faster lane because they feel lower risk.

Look at what CrowdStrike has since committed to in its preliminary post-incident review and root cause analysis: a staggered deployment starting with a canary, monitoring to guide phased rollouts, deployment rings that content must pass through before reaching production, additional validation and bounds checking, and new controls that let customers decide how Rapid Response Content is deployed to their fleets.

None of that is exotic. It is standard progressive delivery, applied to a class of change that had been treated as too small to need it.

Machines are about to write a lot more of your changes

In June, Anthropic released Claude 3.5 Sonnet and reported that in an internal agentic coding evaluation, where the model fixes bugs or adds features to open source codebases from plain-language descriptions, it solved 64% of problems, against 38% for Claude 3 Opus. Whatever the exact numbers on your codebase, the direction is clear: models are getting good enough to produce multi-file changes that pass tests.

That shifts where the risk sits. When a person writes a change, review and the author's own understanding are the main safeguards. When a model writes it, review is often thinner, and the author may not fully understand every line. The safety net has to move to the rollout itself: assume some changes will be wrong in ways tests missed, and make sure a wrong change reaches a few machines, not all of them.

This applies doubly to the fast lanes. AI is very good at generating configuration, policy files, Helm values and Terraform variables, exactly the changes many pipelines push straight to production.

A rollout checklist that does not care who wrote the change

• Every change type gets a ring. Code, configuration, feature flags, content and infrastructure all go through the same staged path. No class of change is too small.
• Canary first, with real traffic. Start with a small, representative slice and wait long enough to see crashes, error rates and latency move.
• Automatic halt. Define the health signals that stop a rollout before you start it, and let the pipeline stop on its own.
• Rollback in one step. Practise reverting each change type. If a bad change can leave machines unable to boot or reach you, your rollback plan cannot depend on them doing either.
• Validate inputs, not just outputs. Schema and bounds checks on generated config catch the class of error tests often miss.
• Give downstream teams control. If you ship to customers or internal platforms, let them choose their ring and their timing.
• Label AI-authored changes. Not to block them, but so you can see their failure rate and tune how far they go before a human looks again.

The regulatory backdrop is changing too

On August 1, the EU AI Act entered into force. It sorts AI systems into risk tiers, from minimal risk with no obligations up to high-risk systems that must meet requirements including risk mitigation and human oversight, with rules for general-purpose AI models applying in 12 months. Most internal coding assistants will not be high-risk systems. But the direction of travel is clear: regulators expect you to show how AI-produced outcomes are controlled, and a documented, staged release process is one of the easiest things to show.

What we recommend

This month, list every path by which a change can reach production, including the quiet ones: config pushes, flag flips, content updates, scripts run by hand. For each, write down its rings, its halt conditions and its rollback. Any path without all three is where your next incident starts, whether a person or a model writes the change.

Then run a rollback drill on the scariest one.

If you would like help building progressive delivery into those paths, the HAZL Solutions team is happy to talk it through.