Out of Resources
Author: jason
Date: 2011-04-29 12:28:09
Category: Technical

Right around the time Service Pack 1 for Windows 7 came out, which was deployed with minimal testing (*sigh*), we began experiencing Resource Exhaustion, Low Memory, Low System Resources, Out of Memory, display monitors going black, Office 2010 claiming not to be licensed and other unusual sporadic problems. This was happening on dozens of computers, across 3 hardware platforms. All were running Windows 7 Enterprise 64bit with Office 2010 32bit and all recently received Service Pack 1. They were also updated with the newest versions of DeviceLock and McAfee. DeviceLock is software that controls what can be plugged into the computers. McAfee is a virus scanning engine. We are also using GuardianEdge, hard drive and removable storage encryption software.

What was causing the problem?

I started my troubleshooting with a jump out to Mark Russinovich's Blog and the Sysinternal pages at Microsoft, along with copious searches on TechNet and Bing to see if other people had seen this problem. I used Process Monitor, Process Explorer, RAM Map, VM Map and of course Task Manager, to start looking at what was happening. Links: Sysinternals, Mark's Blog

I also jumped out and grabbed SvcHost Viewer to see what was hiding in the service host (svchost.exe) processes. Link: SvcHost Viewer

After a few days of monitoring, playing around with Page Files and disabling various services I was getting nowhere. I was no closer to the problem than when I started. Task Manager would show the Physical and Commit levels reaching their max, but I couldn't tell what was consuming those resources.

So I jumped back out to Microsoft and downloaded Poolmon. Maybe my problems were at a lower level. I downloaded the Windows Driver Kit and copied/ran Poolmon.exe on the affected systems. Links: Windows Driver Kit, Poolmon, Using Poolmon

I started by taking screenshots of Poolmon and the Task Manager every 30 minutes or so. Note: I didn't have to enable Tag Mode on Windows 7. I sorted by Byte Usage by pushing B on the keyboard. After a few hours I had a good suspect. A process called DLDR was rapidly increasing in size. After a few reboots, another day of watching, and disabling a few services, I had found the problem.

DLDR happened to correspond to the DeviceLock Driver service. Disabling the service and rebooting revealed no DLDR running in Poolmon. Enabling the service and rebooting showed DLDR gobbling up resources at an alarming rate. On a 2GB RAM computer, running Windows 7 x64 with and without Service Pack 1, DeviceLock was running the system into the ground after 8-10 hours. The process never held steady, never reduced, but continued to climb.

These screenshots were taken all within a 30 minute period.

I contacted the company, opened a ticket, and within a day had a new version of DeviceLock to test. I monitored the new version for 2 days and it all looked good. The process held fairly steady. Or least it didn't eat resources so fast as to grind a machine to a halt.

All is well, until something else breaks.

jason @ jasonthomasfrance.com - www.masterstationlog.com - copyright 2009