CrowdStrike's Faulty Falcon Update: A Detailed Technical Root Cause Analysis
Cybersecurity giant CrowdStrike has released a comprehensive technical root cause analysis detailing the events that led to a problematic Falcon sensor update on July 19, 2024. The incident caused system crashes for some Windows users and prompted a swift response from the company.
The investigation shows that the problem came from a complicated interaction of factors within CrowdStrike’s Rapid Response Content delivery system.
At the core of the problem was a mismatch between the number of input fields expected by the sensor’s Content Interpreter and those provided by a new Template Type introduced in February 2024.
According to the report, the IPC (Interprocess Communication) Template Type was designed to expect 21 input fields, but the sensor code only supplied 20. This discrepancy went undetected during the development and testing phases, partly due to the use of wildcard matching criteria in the 21st field during initial deployments.
The issue occurred when a new version of Channel File 291 was deployed on July 19, introducing a non-wildcard matching criterion for the 21st input parameter. This triggered an out-of-bounds memory read in affected sensors, resulting in system crashes.
CrowdStrike has outlined several key findings and corresponding mitigations:
- Implementation of compile-time validation for template-type input fields
- Addition of runtime array bounds checks in the Content Interpreter
- Expansion of Template Type testing to cover a wider variety of matching criteria
- Correction of a logic error in the Content Validator
- Introduction of staged deployment for Template Instances
- Provision of customer control over Rapid Response Content updates
The company has engaged two independent third-party software security vendors to conduct further reviews of the Falcon sensor code and its end-to-end quality process.
CrowdStrike emphasized that as of July 29, approximately 99% of Windows sensors were back online compared to pre-incident levels. A sensor software hotfix addressing the issue is scheduled for general availability by August 9, 2024.
CrowdStrike has hired two independent third-party software security companies to further review the Falcon sensor code for both security and quality assurance.
