This will be a short’ish post because I’m still trembling from the trauma…
I applied Windows Updates to a client’s production servers yesterday morning. Normally I wait an extra week to give things a chance to “shake out” and get tested in Dev (and other client’s servers) before I apply updates to production, but this time I saw KB2992611 in the announcements and wanted it installed ASAP.
This security update resolves a privately reported vulnerability in the Microsoft Secure Channel (Schannel) security package in Windows. The vulnerability could allow remote code execution if an attacker sends specially crafted packets to a Windows server.
Things were fine yesterday (low load) but once the load started ramping up today the web server was pretty much at 100% CPU – most of it in LSASS.EXE. Everything was very slow and painful. Lots of research, hours of trial and error – including rolling back yesterday’s updates – to no avail. Very frustrating.
As spirits were plummeting, Hans, the client’s resident genius, found this “Microsoft does it again” article. We then realized we should be removing this patch from ALL tiers of the application, not just the front-end. Removed it from the database server and middle app server and then the web server’s load dropped back to normal range. Time for a beer.
Moral of the story? Don’t always focus on the server with high CPU. Look at all the dependencies, especially when you know/suspect you have a bad update in the mix.