CrowdStrike has released a post-incident review (PIR) regarding a faulty update that incapacitated 8.5 million computers. The issue is attributed to the testing software.
The software glitch failed to properly verify the content update that was distributed to millions of machines on Friday. CrowdStrike has pledged to thoroughly test updates of its products, improve error handling, and implement phased rollouts to avoid a repeat of this disaster.
CrowdStrike's Falcon software is utilized by companies worldwide to combat malware and security breaches on millions of Windows computers. On Friday, CrowdStrike rolled out a configuration update for its product designed to "collect telemetry on potential new threat methods." Such updates are issued regularly, but this particular one caused a Windows failure.
CrowdStrike typically delivers configuration updates in two distinct ways. There is the Sensor Content that directly updates CrowdStrike Falcon operating at the Windows kernel level. Additionally, there's Rapid Response Content that modifies the behavior for detecting malware. A small 40 KB file of Rapid Response Content triggered the issue on Friday. Last week, CrowdStrike released two Rapid Response updates, referred to as template instances by the company.
"Due to an error in the content validation tool, one of the two template instances passed verification despite containing problematic data," CrowdStrike notes.
While CrowdStrike conducts both automated and manual testing, it was nevertheless insufficiently thorough. The deployment of new template types in March instilled "confidence in the validations performed in the Content Validator," thus CrowdStrike seemingly assumed that the rollout would not cause issues.
"This unexpected exception was not handled properly, leading to a Windows operating system crash (BSOD)," CrowdStrike explains.
To prevent a recurrence, CrowdStrike vows to enhance the testing of Rapid Response Content through local developer testing, content updates, rollback testing, and stress testing. Furthermore, CrowdStrike will conduct stability and content interface testing for Rapid Response Content and update its cloud validation tool.
Source: The Verge
Comments (0)
There are no comments for now