A flawed security update by Crowdstrike is causing BSOD for windows computers wordwide.

It seems to be caused by csagent.sys
The workaround Steps are:

Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/

Yet another example of "security appliances" being the actual root cause of major system outages/vulnerabilities 🙄

Snakeoil FTW!

Also this incident shows nicely how we have built tons of "critical" systems that all depend on a few single points of failure, rather than focusing on decentralized/federated infrastructures that can provide the required resilience. What could possibly go wrong?

BTW: It is funny how this has been reported as a "worldwide Internet outage" in the mainstream news 😂

a few single points of failure

and automatic updates! Back in the day any update from anyone was first tried on test system(s) before being let loose. With many systems now you have no choice - updates are applied automatically without you having any real control over them. As for knowing what changes these updates make before they are made, well.....

There are reports many of the affected machines are unable to boot at all.

It is funny how this has been reported as a "worldwide Internet outage" in the mainstream news

This is not surprising one whit, very few journalists have even a passing understanding of tech, they are not hired for their computing expertise.

This is nothing surprising to me, this industry push to having everything online with automatic updates funneled through a single source makes the entire IT edifice shaky and vulnerable to a single bad actor being deliberately devious.

It is because MS dismantled their QC team. (Some time ago.) I’m actually surprised something like this hadn’t happened before now.

There is a surprising amount of stuff on YouTube about it, if you are interested. Several ex-MS employees make all kinds of useful videos about it.

Fedora Linux has SilverBlue in which the OS files are immutable. They can be updated with git style transaction / commit /rollback called ostree. Apps are preferably installed /updated via flatpack. One can also have containers called Toolboxes with a choice of OS plus whatever else one wishes to install into them, which is great for developing / testing a particular environment.

There is also Kinoite which is similar to SilverBlue, but with a KDE desktop, for those aren't wildly enthusiastic about gnome.

I guess the question still remains about whether a bad executable can get into an update, hence the need for QC and testing.

Note that it seems the source of this bug was not Windows itself, but rather an automatically updated kernel module by CrowdStrike. It had nothing to do with Microsoft, and I don't even know if the update went through Windows Update.

There's an interesting paradox in how so many people using the same security vendor increases each client's individual reliability because the vendor gets more experienced more quickly and so presumably likewise increases the quality of the product, but decreases the global reliability because the vendor becomes a single point of failure.

It had nothing to do with Microsoft

I've seen and heard many people blame MS for what a 3rd party vendor, CrowdStrike, did. Similar to blaming Boeing when an airline doesn't do proper maintenance on their planes and something goes wrong.

MS and Boeing aren't totally immune to making mistakes.

Windows has the option to do other MS products updates via Window Update, recently Visual Studio was added to that.

https://devblogs.microsoft.com/visualstudio/automatically-install-visual-studio-security-updates-through-microsoft-update/

I don't see any option to do 3rd party updates via Windows Update. CrowdStrike is definitely 3rd party, not MS or Windows.

A company might develop software for multiple different systems. When they make an update of their software for Windows they might call that a "Windows update". If they say "something went wrong with a Windows update" it's easy to see how that could be misinterpreted to mean that something went wrong with an official "Windows Update" from Microsoft.

MS provides an 'official' way to produce a BSOD....


They offer a BSOD screen saver!
https://learn.microsoft.com/en-us/sysinternals/downloads/bluescreen

Great fun - not.

Disclaimer. No computers were harmed during it's usage.

I don't see any option to do 3rd party updates via Windows Update.

Third party device drivers are definitely distributed over Windows Update. It wouldn't be too out of the ordinary to distribute a kernel module over it, although it probably wasn't the case.

Third party device drivers are definitely distributed over Windows Update

Yes - but those drivers are delivered from MS who has been given those drivers by the 3rd party. Windows update doesn't provide any updates from a 3rd party site.

https://www.youtube.com/watch?v=wAzEJxOo1ts gives one view on how the faulty update got in without proper QA and why it was so noticeable.

Interesting re the kernel driver p code interpreter to by-pass having to get each update validated. Only the driver interpreter is validated and signed.

I did something even more hardcore once. It was a driver that received a buffer through an IOCTL, loaded it into executable memory, and ran it. Only an administrator can call that IOCTL, so if something malicious has administrator rights, you're screwed anyway.
Due to a bug in Microsoft's automated tests, we were never able to complete the process to get an extended validation signature. I want to believe it wouldn't have been approved if we had, but I'm also not that optimistic.

far as the blame game goes, the tinfoil hatter in me wonders if its partly intentional -- you blame someone who has more money if your plan is to sue someone to recoup your losses. If they can pin this on M$, even partly ....

I wouldn't normally go there but the last decade has shown this kind of thing to be going on, eg with the idea the gun makers are responsible for shooters (implying long term that car makers are responsible for the vegas strip sidewalk driver and that pressure cooker makers are responsible for the boston marathon bomb and so on). The idiots that do mass injury don't have any money... so they push it up the chain to get someone who does... its almost always about the money, and the older I get, the more I see of that. And there is a constant willful ignorance in the USA at least, where people who know better (news/media folks who have repeatedly been educated on the subjects) continue to push half-truths along the same lines.

I am not normally one to fall into conspiracy theory nonsense, and I even accept I could be seeing something where there is nothing, but this smells. Too many people who know better have tied M$ to this repeatedly.

I can't speak for the accuracy of this, but what I've heard is that some insurance policies require software like CrowdStrike (or perhaps specifically CrowdStrike) to be installed on all insured computers, so if you want someone to blame, insurance companies seems like a good target.

EDIT: Heh. I just did a little calculation. Apparently 8.5M computers were affected. Let's say learning how to fix this, rebooting the machine, and deleting the offending files, takes someone ten minutes. Let's not count how long it takes to actually get in front of the machine, let alone to deal with the aftermath. A little malformed file cost 160 man-years.

MS are now getting in their excuses (sorry defence) early - they're now blaming the EU for allowing the update to cause the IT meltdown!!!

https://www.youtube.com/watch?v=ZHrayP-Y71Q

An update to the previously linked video from David Plummer YouTube Dave's Garage.

Some great quotes:

"A bunch of IT guys with a disk in their hands"

With reference to the Tylenol scandal: " A tamper proof kernel"