I was going to title this post “An Incorrect Server Time can Crash Your ASP.NET 2.0 Applications!” but I thought that would be too sensational… But, it’s true, my ASP.NET 2.0 apps pooped out due to a server time being off. . Here’s what happened:
The other day we started seeing one of our two web servers crashing. We recently deployed our new public site onto ASP.NET 2.0, and everything had been working fine up until this point. The strange thing about this server crashing was that both servers are identical hardware, had the same codebase deployed, and are load balanced with a Cisco CSS hardware. Why one server would begin to fail and not the other was mysterious.
Well, I went looking in the event log and saw this error, at regular intervals, every 5 minutes. It turns out that after a few of these, the server would crash, and we’d get “Service Unavailable” when browsing to that server. This is what the event log looked like:
Source: .NET Runtime 2.0 Error Reporting
Event Id: 5000
Description: EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 microsoft.web.services2, P5 2.0.3.0, P6 42177cce, P7 eb1, P8 47, P9 e3hnjzi4pzb2exb3atbffdtvqefg3pm4, P10 NIL.
What a great description, huh? Anyhow, I did some searching and found the following KB article that let me know what was causing the crash:
Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0
When an unhandled exception is thrown in a Microsoft ASP.NET-based application that is built on the Microsoft .NET Framework 2.0, the application unexpectedly quits. When this problem occurs, no exception information that you must have to understanding the issue is logged in the Application log.
Okay, nice feature. Anyhow, I knew that WSE was somehow involved (due to the web.services2 in the description) and that I had an unhandled exception in code somewhere (sloppy on my part). I suspected a regular process since the errors were so periodic, and we do indeed have some processes that go off every 5 minutes and do some WSE stuff. Well, finally with my trusty copy of SysInternal’s DebugView (which will show you Trace.Write, and Debug.Write output) I tracked it down to this error: A WSE message failed to decrypt because it was “sent in the future.” Sent in the future? Cool! No wait, that’s impposible. I checked and sure enough, the server’s time was off by a minute.
So the time was off, the message wasn’t decrypted, WSE threw an exception, my sloppy code failed to handle it, and bang! Dead server.. I thought I’d pass this along, this was the strangest error fix I’ve had in a while!
-Brendan