In the last method calls post we looked at how a normal method call works in the CLR. I had said that things work as advertised unless code pitching happens. In another previous post Performance Measurement, I made the following statement …
The only really odd bit of code here is the static constructor which insures that our timer methods have been called, once they are called we can assume that they are JIT’ed (even with this bit of code we can still not be assured that our code will be pre-JIT’ed as code pitching (a term I will save for a later post) could occur in between .. but it’s fairly likely)
Code pitching is a fairly obvious but extremely useful concept that can be combined with a JIT compiler. The basic premise of code pitching is that since we have metadata (IL) representing our code, and we have a process (JIT compilation) that can convert our metadata representing our code into executable code; we do not need to keep our executable code in memory at all times. There are some other bits of magic that have to happen in order to allow for this but luckily the CLR manages them for us.
The commercial JIT does not at this time use code pitching, but way back when 1.0 was still in beta there was a JIT that used code pitching in the release. It was known as econojit and is currently used in Rotor (SSCLI) and the compact framework.
If we look back to the last method calls post we can see that the calls are made indirectly through the use of a table. When the code has not yet been JIT’ed this table points to a thunk that will JIT the code and then alter the table so future calls will directly call the JIT’ed code. In code pitching we do the exact opposite of this process, we take and set the table back to the thunk that will perform JIT compilation, this allows us to throw away the memory holding the code we have already compiled.
This may not sound like much of a gain but consider how many methods you have in your application that are only called on start up. You keep the native versions of these methods in memory for the entire life of your application simply in case you happen to call them again. In many cases you would be better served to release the memory to use it for more important items.
Another reason why code pitching becomes important (especially in environments like the compact framework) is that some processors, especially RISC processors will have much larger representations of the code than would be expressed in IL. This is not really a problem with x86 code as it is usually roughly the same size or smaller than the equivalent IL but a great example can be seen in the IA32 processor where IBM reported the code size to be on average 6-8 times larger than the equivalent java byte code.
Code pitching also allows the CLR to put a limit on how much compiled code it can hold at any given moment. For a large application that is running on a busy desktop this can help your application to play nice with its memory usage as it gives the CLR a bit more to work with when pressure is to be places on it.
Of course the major benefit of code pitching has nothing to do with it being a memory amortization; it can also be a JIT amortization. The JIT optimizer has a requirement of being very fast, this means that it can often times not do some more costly optimizations that may create better code (this is not a problem with say C++ because the compilation is being done up front). In this scheme the JIT is really one size fits all. Why not have the JIT optimize the way we micro-optimize?
By using code pitching we can use both an “average” JIT and slower “optimized” JIT. At first we compile the methods using the average JIT. The average JIT marks whether or not it thinks the code is a good candidate for further optimizations. We track the usage of the “hot spots” as the system runs. If a method is being called a lot and it has been marked as a method that could benefit from further optimizations we can pitch the code and mark it for compilation with our better optimizing JIT.
This allows us to have the JIT optimize code in a very similar method to how a developer would micro-optimize code. It amortizes the cost of the better optimization by only applying it to methods that are frequently used. Code pitching in both cases is all about amortization…
Code pitching is not a new concept; the jalapeno project was initiated in 1997. This is unfortunately one of the places where the .NET runtime is way behind the Java world and it is an important one. My guess is that it is only a matter of time before this type of amortization makes it into the commercial JIT.
References worth reading:
To Jit or not to Jit: The Effect of Code Pitching on the Performance of the .NET Framework
The Jalapeno Virtual Machine
The Jalapeno Dynamic Optimizing Compiler for Java
Adaptive Code Unloading for Resource-Constrained JVMs
Profile-driven Code Unloading for Resource-Constrained JVMs