Lessons from the CrowdStrike outage.

What follows is a first-person account of the response from inside one of the affected environments, and what the incident revealed about the gap between we have security tools and we can recover.

The night

It was 0147 EDT, overnight into Friday morning. I was about to call it a night when the monitoring dashboards started changing color, the systems alerts came in, and the phones began ringing.

The unusual stillness of a quiet Thursday gave way to the recognition that something was very wrong, simultaneously, across multiple data centers and continents. This is the texture of modern infrastructure failure: not a single point of pain spreading outward, but synchronized failure across systems that share a common dependency you didn't fully appreciate until that moment.

The response

When the incident response protocol activated, the work split into three immediate streams: identify the pattern, contain the damage, and structure the recovery.

I revisited the Planning Considerations for Cyber Incidents guide from CISA, used in conjunction with our own business continuity, disaster recovery, and incident response plans. The CISA framework gave us a structure for coordinating across teams. Our internal plans gave us the specifics — which systems to bring back first, which dependencies had to be respected, which stakeholders needed updates and how often.

The pattern emerged quickly: every affected system had the CrowdStrike Falcon agent and had received the bad update. The fix was known within hours of CrowdStrike's public statement, but the operational work of executing it across thousands of endpoints — many of them in BitLocker recovery loops, many requiring physical or console access — was the actual challenge. There is no automation for "boot into safe mode and delete a specific file" when the systems can't network to receive instructions.

What got us through wasn't tooling. It was the team — application, infrastructure, and security working as a single response unit, with clear decision authority, structured recovery procedures developed and validated in flight, and disciplined communication that kept stakeholders informed without flooding the responders.

By the time CrowdStrike issued its formal statement, we had achieved near-total restoration across multiple data centers. Many users had not yet had their first cup of coffee.

What the incident actually revealed

The post-incident analysis surfaced several things worth saying out loud, because they apply far beyond this specific event.

1. Lesson 01
   Automatic vendor updates are a single point of failure most organizations have never properly examined.

   The same mechanism that protects you from the latest threat is the mechanism that crashed nine million machines. Automatic updates are not inherently wrong. But the governance around them — testing windows, staged rollouts, ability to pause or roll back — is often weaker than the protection benefit assumes.
2. Lesson 02
   The blast radius of a single dependency is rarely understood until it fails.

   Most organizations had no clear answer to the question "which of our systems run Falcon?" until they were forced to find out. The same is true of any agent that runs at the kernel level or has broad system access — Falcon, similar EDR products, identity agents, backup agents. The list of "things that can take down everything if they break" is longer than most CMDBs reflect.
3. Lesson 03
   Recovery is a capability, not a plan.

   A documented recovery process and an executable recovery process are different things. The organizations that recovered fastest had recently practiced the underlying work — manual remediation, BitLocker key retrieval, console access at scale, communication under pressure. The ones that struggled most had documented those processes but never proved them.
4. Lesson 04
   Communication discipline matters as much as technical execution.

   Internal teams need updates often enough to coordinate but not so often that responders are spending their time writing status reports instead of fixing systems. Executives need translation, not raw incident detail. Customers need honesty about what they can expect and when. None of that is improvised well at 3 AM.

What this means going forward

The CrowdStrike incident wasn't a cyberattack, but it taught the same lessons one would have. The defense layer assumed: every adversary now knows how a major outage actually unfolds in a real environment. Response patterns, coordination structures, recovery times, communication cadences — all observable. State-sponsored actors and organized criminal groups will not have missed this.

For organizations still operating in what I call Incident Response 1.0 — reactive, document-driven, and rarely tested — the gap between policy and capability is the real exposure. Several practical shifts move that gap in the right direction:

• Treat update governance as a security control, not an operational nuisance. Stage rollouts. Maintain pause-and-rollback capability. Verify it works before you need it.
• Map kernel-level and high-trust agents explicitly. Know what runs where, and what fails if any one of them fails.
• Practice manual recovery at small scale, regularly. The skills atrophy if not exercised, and you cannot manufacture them in the middle of an event.
• Build incident communication as its own muscle. Cadence, audience, content — all rehearsed, not improvised.
• Treat post-incident review as a learning function, not an accountability function. Blameless reviews surface the actual lessons. Blame-driven reviews surface defensive narratives.

The fragility revealed by this outage isn't going away. The dependencies that produced it are deeper now than they were two years ago, not shallower. The organizations that come through the next one well will be the ones treating resilience as an operating discipline — built before the incident, drilled regularly, and continuously improved — rather than as a document.