Greg Young [MVP]

In my last 2 posts we spent a bit of time looking at asynchronous socket buffer management and Continued. In dealing with these two posts I came across a few things, which appear to be problems in the framework.

I promised that I would be a good boy blog more so here we go ... The first problem I ran into is the # of pinned objects counter that the CLR exposes.

Using the code provided in the second post with 500 clients, run the server (release mode). Bring up perfmon and point it at the server for “# of pinned objects” located in CLR memory. It should show 7 pinned objects.

Now break the server and load SOS. Run !GCHandles

!GCHandles

GC Handle Statistics:

Strong Handles: 28

Pinned Handles: 7

Async Pinned Handles: 0

Ref Count Handles: 0

Weak Long Handles: 46

Weak Short Handles: 9

Other Handles: 0

Statistics:

      MT    Count    TotalSize Class Name

790f7698        1           12 System.Object

79153fd4        1           16 System.Threading.RegisteredWaitHandle

791540c8        1           20 System.Threading._ThreadPoolWaitOrTimerCallback

791138d0        1           24 System.Threading.ManualResetEvent

790f90cc        1           28 System.SharedStatics

790f8688        1           72 System.ExecutionEngineException

790f85e4        1           72 System.StackOverflowException

790f8540        1           72 System.OutOfMemoryException

790f92b8        1          100 System.AppDomain

790f8c24        2          112 System.Threading.Thread

790fa154        5          120 System.Reflection.Assembly

7a77273c        4          144 System.Net.Logging+NclTraceSource

790f8790        2          144 System.Threading.ThreadAbortException

7a772810        4          160 System.Diagnostics.SourceSwitch

79104da8        4          192 System.Reflection.Module

790fb43c        7          252 System.Security.PermissionSet

790ff76c       46         3496 System.RuntimeType+RuntimeTypeCache

7911eb1c        7        17456 System.Object[]

Total 90 objects

So it is correct so far, the 7 handles match our shown 7 pinned handles. Start the client and let all 500 connect (it will tell you when they are done). Watch perfmon until you see a pattern like in the picture here (note it goes up and then falls back down).

When it has come back down to 0 (will take about 45 seconds) break into SOS again.
 

!GCHandles

PDB symbol for mscorwks.dll not loaded

GC Handle Statistics:

Strong Handles: 27

Pinned Handles: 7

Async Pinned Handles: 501

Ref Count Handles: 0

Weak Long Handles: 3

Weak Short Handles: 11

Other Handles: 0

Statistics:

      MT    Count    TotalSize Class Name

790f7698        1           12 System.Object

790f90cc        1           28 System.SharedStatics

790f8688        1           72 System.ExecutionEngineException

790f85e4        1           72 System.StackOverflowException

790f8540        1           72 System.OutOfMemoryException

790f92b8        1          100 System.AppDomain

790fa154        5          120 System.Reflection.Assembly

7a77273c        4          144 System.Net.Logging+NclTraceSource

790f8790        2          144 System.Threading.ThreadAbortException

7a772810        4          160 System.Diagnostics.SourceSwitch

79104da8        4          192 System.Reflection.Module

790ff76c        3          228 System.RuntimeType+RuntimeTypeCache

790fb43c        7          252 System.Security.PermissionSet

790f8c24        6          336 System.Threading.Thread

7911eb1c        7        17456 System.Object[]

79153be0      501        34068 System.Threading.OverlappedData

Total 549 objects

The perf counter is telling us there are 0 pinned objects but this not seem to agree with SOS. This is very disconcerting since one of the things we should be watching during debugging is our # of pinned objects to help watch out for heap fragmentation but it gets worse.

This counter only shows us the GCHandle generated pinned objects, these are not the only pinned objects. The Marshaller also pins objects when it handles calls to unmanaged code (like when we call into say the unmanaged socket libraries). In this particular example there are about 3-4 times as many pinned objects than we can see. You will notice that the pins here are on OverlappedData objects, my actual buffers are also pinned but we can’t see that they are pinned from here. In doing some research there is apparently no way to view the objects that the marshaller has pinned so in order to gauge what’s going on we are bound to only looking at resulting heap fragmentation and trying to minimize it.

This can be seen in trying to compare the two versions of the server (with and without buffer management). If you run a !GCHandles where I ran the heap commands in the previous post you will notice that the same number of GCHandles are listed for each version yet the buffermanager version has far less heap fragmentation. This is because the pins on the actual byte arrays are done by the marshaller as such we can't see them but can only view their effect of causing heap fragmentation. There are other things which are being pinned in this process which is why both versions show heap fragmentation (3.5 includes a new version of this which pins far less objects, perhaps we should give it a whirl in another post?)