Problem : How to fix IIS Worker Process’ Virtual Bytes from constantly increasing
We have been experiencing both “OutOfMemory” and “memory cannot be allocated” errors on our production website. This has been occurring up to 2 or 3 times a day over the past month.
Further investigation revealed that the Virtual Bytes used by the worker process at the time of the errors occurring was about 1.7 GB, yet the private bytes value sat comfortably at about 500MB.
Our webserver is powerful, 16gigs of memory, 4 dual core xeon processors. Windows Server 2003 Enterprise edition running IIS6 in native mode. We run a few different web apps on the server, each in it’s own Application Pool.
Unfortunately for a performance benefit we’ve used inProc session state, so we cannot use a web garden at this stage. (Although I’m starting to think this should be our next step to boost both performance and reliability) and I’m reluctant to switch on memory limit recycling as that would kill all our clients inProc sessions whenever it gets recycled. Yet we’re manually resetting IIS anyway.
When the app is first loaded, virtual bytes sit at about 750mb and private bytes about 200mb. During the day the virtual bytes on the worker process steadily rises until the errors start occurring (at about 1.7GB) We rely quite heavily on the server taking quite a load, large cache entries as well as small bits of session information that we constantly collect. Our app’s requirement is to take as much load from the client and database as possible, which would seem to make it susceptible to memory fragmentation problems.
It would on face value seem as though we’re fragmenting our virtual memory, when we try load a datatable or something quite large, we cannot get a large enough block.
I’ve read a few articles about machines with large amounts of memory in .NET 1.1 believing they had more memory than actual:
ASP.NET Process Model When using the ASP.NET process model, you configure the memory limit in the Machine.config file as follows.
This limit is important to adjust when your server has 4 GB or more of RAM. The 60 percent default memory limit means that the worker process is allocated 2.4 GB of RAM, which is larger than the default virtual address space for a process (2 GB). This disparity increases the likelihood of causing an OutOfMemoryException. To avoid this situation in .NET Framework 1.0, you should set the limit to the smaller of 800 MB or 60 percent of physical RAM. .NET Framework 1.1 supports a virtual space of 3 GB. If you put a /3GB switch in boot.ini, you can safely use 1,800 MB as an upper bound for the memory limit.
IIS 6.0 Process Isolation Model When using the IIS 6.0 process isolation model, you can configure the memory limit as follows.
Open the Internet Information Services (IIS) Manager.
Right click the Application pool.
On the properties, select the Recycling tab.
Unselect the Recycle worker process (in minutes):.
In the Memory recycling section select Maximum used memory (in megabytes)
Set the memory limit to the calculated limit
Note The Memory recycling application pool setting allows you to specify separate limits for physical memory and virtual memory. Physical memory (or a combination of both physical and virtual memory) is commonly used for ASP.NET applications because of the way the common language runtime’s garbage collector works. Virtual memory is commonly used for classic native applications that fragment the heap.
Questions/Discussion
1. Since I’m using Native mode, do I need to specifically set the memory recycling limit to induce the garbage collector to kick in earlier? Currently it is set to off.
2. Is 700Mb virtual memory far too large on startup? Could this be an assembly loading issue? How can I verify this with the CLR profiler – I’ve just began experimenting with this.
3. Is recycling the app pool due to memory fragmentation acceptable with larger apps? It seems like a band aid solution, but might be a standard industry practice for leaky apps?
Solution : How to fix IIS Worker Process’ Virtual Bytes from constantly increasing
I came across this article and then find out the error…
https://docs.microsoft.com/en-us/archive/msdn-magazine/2006/november/clr-inside-out-investigating-memory-issues
If the largest free block of virtual memory for the process is less than 64MB on a 32-bit OS (1GB on 64-bit), the OOM could be caused by running out of virtual memory. (On a 64-bit OS, it is unlikely that the application will run out of virtual memory space.)
The process can run out of virtual space if virtual memory is overly fragmented. It’s not common for the managed heap to cause fragmentation of virtual memory, but it can happen. For example, it might occur if the app creates a lot of temporary large objects, causing the LOH to constantly acquire and release segments.
The !eeheap gc SOS command will show you where each garbage collection segment starts. You can correlate this with the output of !address to determine if the virtual memory is fragmented by the managed heap.
The following are some other typical conditions that can lead to fragmented virtual memory:
Many small assemblies get loaded and unloaded all the time.
Lots of COM DLLs loaded because of COM interop.
Assemblies and COM DLLs are not both loaded in the managed heap. A common scenario that can lead to this is when an ASP.NET site has been compiled with the <debug> config flag enabled. This leads to every page being compiled in its own assembly, possibly fragmenting the virtual memory space enough to lead to OOMs.
It turns out, a colleague had replaced the webconfig with a new version (we normally just leave the old one when deploying) which had compilation mode debug = true.
So far setting this to false has stopped our memory allocation errors. Interestingly though our virtual bytes has remained high – 1.7GB, but is stable.
