Software failures
The validator and the sensor disagreed about the data. Windows machines crashed.
The July 2024 outage began as a mismatch between what a definition promised and what the code supplied. Databases meet the same gap every day.
Published 7 min read
Jump to the SQL version of the check
Photo: CrowdStrike BSOD at LGA, Smishra1, CC BY-SA 4.0, via Wikimedia Commons.
On 19 July 2024, CrowdStrike pushed a content update to its Falcon security sensor on Windows machines. Many of those machines crashed. CrowdStrike's root cause analysis, published on 6 August 2024, says that about 99% of Windows sensors were back online by 29 July at 8:00 p.m. Eastern time, compared with the level before the update.
The headline version of this story is "a bad update". The report is more specific, and the specifics are about data. Two components each held a belief about how many inputs a record would have. The beliefs differed by one. Nothing in the pipeline compared them until real machines did.
The parts, in plain terms
CrowdStrike ships two kinds of detection content. Sensor Content is compiled into the sensor release. Rapid Response Content is delivered from the cloud so the company can respond to new attack techniques quickly, without changing sensor code. The report stresses that Rapid Response Content is configuration data, not code and not a kernel driver.
| Part | What the report says it does |
|---|---|
| Template Type | Predefined fields for detection engineers to use in content. Compiled into the sensor at build time. |
| Template Type Definitions file | Defines each Template Type, including how many inputs it is meant to use. |
| Template Instance | Matching criteria written by detection engineers, as regular-expression content for one Template Type. |
| Channel File | The delivery bundle for Rapid Response Content. This incident involved Channel File 291. |
| Content Interpreter | Sensor code that tests input strings against the regular expressions. |
| Content Validator | Checks channel files against their definition in the Template Type Definitions file. |
Read the last two rows together. The validator checked content against the definition. The interpreter ran content against what the sensor actually supplied. If the definition and the supply disagree, a file can pass the first check and break at the second.
What the mismatch was
In February 2024, sensor version 7.11 introduced a new Template Type for detecting abuse of named pipes and other Windows interprocess communication, called the IPC Template Type. According to the report, the definition said it took 21 input fields. The sensor code that called the Content Interpreter supplied only 20.
That gap sat there, unnoticed, for months. The report gives two reasons. During testing, and in the first IPC content that went out, the 21st field used a wildcard match. A wildcard never needs to look at the value, so the missing 21st input was never read. The first several production deployments of IPC content also used wildcards in that position.
On 19 July 2024, two new IPC Template Instances went out. One used a non-wildcard criterion for the 21st input for the first time. The sensor now had to read a value that did not exist. The Content Validator evaluated the new content, but, in the report's words, it based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs. The result was an out-of-bounds memory read and a crash. The report says this was not an arbitrary memory write issue and that this has been independently reviewed.
The sequence
From definition to crash
February 2024
Sensor 7.11 adds the IPC Template Type. The definition lists 21 inputs; the sensor code supplies 20.
Before 19 July
Earlier IPC content uses wildcards in the 21st field, so the missing input is never read.
19 July 2024
Two new IPC Template Instances ship. One matches on the 21st input. Affected sensors crash on the next IPC event.
25 July
CrowdStrike adds bounds checking to the function that retrieves input strings.
27 July
A compiler patch that validates the number of inputs in a Template Type goes into production.
6 August
The root cause analysis is published.
Why the tests passed
The report is candid about this. Automated testing used a static set of 12 test cases. The channel file those tests used had been chosen by hand and put a wildcard in the 21st field of every Template Instance. So the very tests meant to exercise the new type avoided the one condition that would have shown the problem.
This is a pattern worth recognising. A test can be thorough in volume and still miss the case that matters if every sample shares a convenient property. All of the test data looked the same in the one place where production data would later differ.
The general lesson: one definition, enforced on both sides
The root of the problem is that "how many inputs" lived in two places: the definition file and the sensor code. Whenever a fact lives in two places, someone has to make sure the places agree, and "someone" usually means nobody.
Databases offer this discipline in a form you can use today. A schema is a single definition that both writers and readers must obey. A constraint is enforcement at the moment of writing, not a hope that a reader copes later. Where two systems need to agree, store the agreement as data and check it with a query.
The same idea in SQL
This SQLite sketch models the situation with fictional tables. A type records how many inputs its definition says it has and how many its code supplies. An instance records the highest input position its content reads. A single query finds content that reads beyond what is supplied.
CREATE TABLE template_type (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
defined_inputs INTEGER NOT NULL,
supplied_inputs INTEGER NOT NULL
);
CREATE TABLE template_instance (
id INTEGER PRIMARY KEY,
type_id INTEGER NOT NULL REFERENCES template_type(id),
highest_input_read INTEGER NOT NULL
);
INSERT INTO template_type VALUES (1, 'ipc', 21, 20);
INSERT INTO template_instance VALUES (1, 1, 20), (2, 1, 21);
-- Content that reads past what the code supplies
SELECT i.id, t.name, i.highest_input_read, t.supplied_inputs
FROM template_instance i
JOIN template_type t ON t.id = i.type_id
WHERE i.highest_input_read > t.supplied_inputs;
-- returns instance 2Better than a query is to make the disagreement impossible to store. If the two numbers must be equal, say so:
CREATE TABLE template_type (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
defined_inputs INTEGER NOT NULL,
supplied_inputs INTEGER NOT NULL,
CHECK (defined_inputs = supplied_inputs)
);
INSERT INTO template_type VALUES (1, 'ipc', 21, 20);
-- Error: CHECK constraint failed: defined_inputs = supplied_inputsThat mirrors the first mitigation in the report: validate the number of inputs in a Template Type when the sensor is compiled, so the mismatch fails the build instead of shipping. The constraint is a stand-in for that compile-time check.
Test the edge, not the sample
The second lesson is about test data. If a field can be a wildcard or a value, test both. If a column can be NULL, empty or populated, include all three. If a list can have zero, one or many entries, test each. CrowdStrike's fix was of this kind: automated tests with non-wildcard criteria for every field of every Template Type, required for future types.
-- Which values does each field actually take in the test data?
SELECT COUNT(*) AS rows_total,
SUM(field_21 IS NULL) AS nulls,
SUM(field_21 = '*') AS wildcards,
SUM(field_21 IS NOT NULL AND field_21 <> '*') AS real_values
FROM test_instances;If the real-values count is zero, your tests have never exercised the field. That is a finding before any code runs.
Release in rings
The sixth finding in the report concerns deployment. Each Template Instance should be rolled out in stages. The company describes canary testing, then promotion through successively wider rings, with extra "bake-in" time between rings while telemetry is gathered, and rollback if problems appear. It also added controls so customers can choose where and when Rapid Response Content is deployed.
You can apply the same shape to a database migration or a data load. Run it on a copy. Then on one small tenant or region. Then on a fraction of production. Look at error rates between steps. A change that fails in the first ring harms a few people instead of all of them.
- 01Validate the shapeCheck the number, type and range of fields against the definition the reader uses, not a similar test definition.
- 02Test both kinds of valueInclude the wildcard, the NULL, the empty list and the real value in your samples.
- 03Release to a small ringApply to a copy, then a small group, then wider groups with time to watch between.
- 04Keep a way backKnow which version to restore and how long it takes.
What this does not show
- It does not say a database constraint would have prevented this outage. The failing components were in a security sensor, not a SQL database.
- It does not rank the causes. The report itself describes a combination of issues rather than a single one.
- It does not cover how customers recovered machine by machine. Recovery for many organisations involved manual steps and took days.
Compare two definitions with a query
The failure came from two places each holding a version of "what the inputs are". A cheap check is to store both lists and ask for the difference. This SQLite sketch uses JSON arrays and json_each. It was run and returns the one field that is defined but not supplied.
CREATE TABLE type_def (type TEXT PRIMARY KEY, fields TEXT); -- JSON array
CREATE TABLE type_supplied (type TEXT PRIMARY KEY, fields TEXT);
INSERT INTO type_def VALUES ('ipc', json_array('a1', 'a2', 'a3'));
INSERT INTO type_supplied VALUES ('ipc', json_array('a1', 'a2'));
SELECT d.type, j.value AS defined_but_not_supplied
FROM type_def d, json_each(d.fields) j
WHERE j.value NOT IN (
SELECT value FROM type_supplied s, json_each(s.fields)
WHERE s.type = d.type);
-- ipc | a3Run a check like this wherever a definition and an implementation live apart: API schemas and handlers, database migrations and ORM models, config files and the code that reads them. It will not catch everything, but a mismatch of this kind should never reach production.
Model a staged rollout
CrowdStrike's report describes canary testing and successive deployment rings with bake-in time between them. Even a small table makes the rule explicit: a change may only move to the next ring when the previous ring has run without errors.
CREATE TABLE host (id INTEGER PRIMARY KEY, name TEXT, ring INTEGER);
CREATE TABLE rollout (change_id TEXT, ring INTEGER, started TEXT, errors INTEGER);
-- Hosts per ring: 0 = canary, 1 = early, 2 = everyone else
SELECT ring, COUNT(*) FROM host GROUP BY ring;
-- Next ring that change c1 may go to: only after a clean earlier ring
SELECT MIN(ring) FROM host
WHERE ring > (SELECT MAX(ring) FROM rollout WHERE change_id = 'c1' AND errors = 0);The point is the discipline, not the SQL. The rule "promote only after a clean ring" is written down and checked by a query, so it does not depend on someone remembering under pressure.
Questions that come up
- Was this a cyberattack?
- No. CrowdStrike's analysis describes a defect in its own content and sensor code, and says it found the issue was not exploitable by a threat actor.
- Why did a configuration file crash the machine?
- The sensor includes a kernel driver that reads the content. The Content Interpreter read past the end of an input array, which crashed the system. The report notes that the content itself is configuration data, not a driver.
- What is the simplest thing I can do from this?
- Find one place where two systems must agree on a count, a type or a unit. Store the agreement in one place and make the other side check it.
- Can a database constraint replace testing?
- No. A constraint catches stored data that breaks a rule you wrote down. It cannot catch a rule you forgot to write, which is how this mismatch survived.