Tag Archive for: IIS

KB2992611 Breaks More than Web Servers

17 Nov 2014
November 17, 2014

This will be a short’ish post because I’m still trembling from the trauma…

I applied Windows Updates to a client’s production servers yesterday morning. Normally I wait an extra week to give things a chance to “shake out” and get tested in Dev (and other client’s servers) before I apply updates to production, but this time I saw KB2992611 in the announcements and wanted it installed ASAP.

This security update resolves a privately reported vulnerability in the Microsoft Secure Channel (Schannel) security package in Windows. The vulnerability could allow remote code execution if an attacker sends specially crafted packets to a Windows server.

Things were fine yesterday (low load) but once the load started ramping up today the web server was pretty much at 100% CPU – most of it in LSASS.EXE. Everything was very slow and painful. Lots of research, hours of trial and error – including rolling back yesterday’s updates – to no avail. Very frustrating.

As spirits were plummeting, Hans, the client’s resident genius, found this “Microsoft does it again” article. We then realized we should be removing this patch from ALL tiers of the application, not just the front-end. Removed it from the database server and middle app server and then the web server’s load dropped back to normal range. Time for a beer.

Moral of the story? Don’t always focus on the server with high CPU. Look at all the dependencies, especially when you know/suspect you have a bad update in the mix.

A Quick Look at IIS Crypto

22 Apr 2014
April 22, 2014

For many years now – I think about a decade — I have been building and managing Windows Web servers for a SaaS company. First as a contractor, then an employee and most recently as a consultant (tip: relationships matter). While I don’t have it down to an automated science, I do have pretty detailed checklists that I run through as part of each build.

My goal has always been to keep the server configurations consistent yet adapt as each new Windows Server release comes out. My first checklist was for a Server 2000 build. The one I updated last week was for 2012 R2. That’s covering some serious territory.IIS has certainly changed a lot over the years.

Of course, a big part of the builds revolves around security. One aspect of security is managing the SSL protocols and cipher suites offered by IIS. And, more importantly, which ones you do not wish to offer. I’ve documented this over the years and done some automation with batch files, vbs files (I know, I know…) and even .reg (registry import) files.

As I was updating the most recent servers last week I thought I had better do some quick research and make sure my notes were still current. These things date back 5 – 10 years after all. For instance, I know just half a year ago we went through the servers and disabled RC4 at Microsoft’s suggestion. That was never folded into my build notes…

While looking for SSL security updates I stumbled over a very handy little utility: IIS Crypto. I downloaded the tool, tried it on a dev server and then hit it with an external Qualys scan. It worked great – and fixed up a few holes that I had incorrectly plugged over the years (whoops). Now IIS Crypto is part of my toolbox.

IIS Crypto Screenshot

Just download and run it on your web server and choose the “Template” that applies. For me, that’s Best Practices with one click, then a second click to disable RC4 128/128 just to be consistent. Hit apply, schedule a reboot and you’re in good shape. Need to be PCI or FIPS compliant? Those are templates there as well.

There’s even a quick way to scan yourself from Qualys SSL labs.

Very slick. Highly recommended.

From their site:

IIS Crypto is a free tool that gives administrators the ability to enable or disable protocols, ciphers, hashes and key exchange algorithms on Windows Server 2003, 2008 and 2012. It also lets you reorder SSL/TLS cipher suites offered by IIS, implement best practices with a single click and test your website.

IIS: No Monitoring Hits in the Logs

12 Sep 2010
September 12, 2010

A few weeks ago I wrote about how I was changing my IIS web server configurations to (hopefully) better manage memory and App Pools. That’s been working out quite satisfactorily. Yesterday I realized the changes I made can give another nice side effect.

I was preparing to start some maintenance yesterday morning and I wanted to make sure nobody was hitting the development web server. One easy way to do that is to just check the IIS logs and that’s what I did. Open the log, jump to the end and then scroll back up past all the once a minute “is the server up?” monitoring hits … and it suddenly hit me, those monitoring hits don’t need to be there!

It would be nice to be able to open a log file, jump to the bottom and see the last real hit without having to filter through all the monitoring hits. Since I recently put the page the monitors hit in his own Application (and app pool) this is remarkably trivial to do. One mouse click, in fact.

Seems obvious, doesn’t it? Just uncheck “Log visits.” I couldn’t do this in the past since the monitoring page was in the same app / app pool as all the client sites. But now I can — and I like it.

Oh, Did You Want a Timeout?

24 Aug 2010
August 24, 2010

Back in February I mentioned that I was reconfiguring my IIS 6 web servers to shutdown the App Pools after 2 hours of inactivity. That seemed a much better option than the brute force iisreset that I’d been scheduling as a nightly event.

Turns out I wasn’t quite done yet. Here’s a snip of a conversation I had earlier today with my CTO, Hans.

“Ya know, I wish I had some better tools to see how many active users we have across all the sites at a given moment,” I said. “That would be helpful when I want to sneak in a quick change during the day.”

“Well what do you currently do to check?” he asked.

“I just pop open the latest IIS log file, jump to the bottom and see if the most recent entries are from my once-a-minute WhatsUp Gold site monitoring. If the last few entries are from WhatsUp then I know we’ve been idle that many minutes.”

He nodded and we moved onto another issue which resolved around some memory related issues.

I commented, “It seems like this main w3p process never shrinks. It just keeps growing its memory usage. How weird. Come to think of it, I’ve never seen a Windows Event about it shutting down or spinning back up…”

Hans just gave me the look and said, “Didn’t you mention your monitoring process hits that site every minute?”

#facepalm#

“oh yeah… I guess it’ll never hit that 2 hour timeout, huh?” Don’t laugh… If you poll the site every minute don’t expect it to ever go idle!

Today I spent some time fixing that. I’m not sure what the best practices are but I have an approach that seems reasonable.

First, I created a new site with just one page (ping.html). Next, I created a new App Pool called monitoring just like the Default. But instead of a timeout I configured it to restart itself at 1:00 AM nightly. Then I converted that new site to an IIS application using that new App Pool.

IIS App Pool settings

My maint site's IIS app settings

I changed the WhatsUp monitor to use a custom HTTP Content monitor pointed at the new site. Now it tests for content from the ping.html page instead of just seeing if something responds on port 80 so this is probably even a bit better than it was before.

This brought up another small issue though.

Wait! How do I know which w3p process ties to which App Pool?

Now I have more App Pools all running as the same user. How can I quickly tell which process goes to which pool? Easy!

This picture lays it out:

App Pools and w3p processes

On the IIS server bring up a command prompt, navigate to the system32 directory and run:

cscript iisapp.vbs

The output lists the process ID (PID) and name for each w3p.exe process. Problem solved.