Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

Performance Measurement

I promise this is the last ground laying post before I move on to something of consequence but I wanted to get this post out so I can reference it in the future as this will be the test bed used to get performance metrics for code (and my next post is already too long by itself, just sillily long if I include this and the previous post with it)


As I had said in my last post the best place to start optimizing is through direct measurement. Since we are aiming for the fastest execution time; the best way to get there is to measure how fast given code is run. Here is a test harness I wrote some time ago based loosely upon recommendations from the P&P group that can be seen here.  


The first thing we need is a high resolution timer as DateTime.Now.Ticks will not cut it for us. This code uses the QueryPerformanceCounter API in order to provide us with this higher resolution timer. A key difference from the P&P example is that I wanted to be able to reuse my timer for simply timing things as opposed to only working with iterations then passing in 1 as my number of iterations, imho they got that part kind of backwards.








    Class HighResolutionTimer {


        private long m_Start = long.MinValue;


        private long m_Stop = long.MinValue;


        Decimal multiplier = new Decimal(1.0e9);


 


        [DllImport(“Kernel32.dll”)]


        private static extern bool QueryPerformanceCounter(out long lpPerformanceCount);


 


        [DllImport(“Kernel32.dll”)]


        private static extern bool QueryPerformanceFrequency(out long lpFrequency);


 


        public void Start() {


            if (m_Start == long.MinValue) {


                if (!QueryPerformanceCounter(out m_Start))


                    throw new Exception(“QueryPerformanceCounter Failed”);


            } else {


                throw new Exception(“Already Started”);


            }


        }


 


        public void Stop() {


            if (m_Start > 0) {


                if (m_Stop == long.MinValue) {


                    if (!QueryPerformanceCounter(out m_Stop))


                        throw new Exception(“QueryPerformanceCounter Failed”);


                } else {


                    throw new Exception(“Already Stopped”);


                }


            } else {


                throw new Exception(“Not Started”);


            }


        }


 


        public void Reset() {


            m_Start = long.MinValue;


            m_Stop = long.MinValue;


        }


 


        public double NanoSeconds {


            get {


                if (m_Start == int.MinValue)


                    throw new Exception(“Not Started”);


                if (m_Stop == long.MinValue)


                    throw new Exception(“Not Stopped”);


                long Frequency;


                if (!QueryPerformanceFrequency(out Frequency))


                    throw new Exception(“QueryPerformanceFrequency Failed”);


                return (double) (m_Stop – m_Start) * (double) multiplier / (double)Frequency;


            }


        }


    }


Figure 1: Timer


 


The next thing we need is a simple harness to use our new timer so we don’t keep duplicating code.








    public delegate void TestHandler();


    class TestHarness {


         


        public static void Test(string _Name, int _Times, TestHandler _Handler) {


            if (_Handler == null) throw new ArgumentNullException(“_Handler”);


            _Handler();


            HighResolutionTimer t = new HighResolutionTimer();


            t.Start();


            for (int i = 0; i < _Times; i++) {


                _Handler();


            }


            t.Stop();


            double total = t.NanoSeconds;


            double average = total / _Times;


            Console.WriteLine(“Test : “ + _Name + ” took “ + total + ” ns, average ns = “ + average);


        }


 


        static TestHarness() {


            HighResolutionTimer t = new HighResolutionTimer();


            t.Start();


            t.Stop();


        }


    }


Figure 2: Test Harness


 


Note that a nicer interface might be to build a “Test” object that contains the name of the test, the iterations to run and the delegate, I leave this as an exercise to the reader to implement :)


As you can see, the test harness is given a delegate (the method to test).  It calls this method _Times times and outputs the total time and the average time for each operation. It is important to note that I have another test harness that I use for less deterministic things that calculates time for each operation and also gives me a standard deviation (you should use a test like this when testing things that are highly variable such as reading data from a database). The only really odd bit of code here is the static constructor which insures that our timer methods have been called, once they are called we can assume that they are JIT’ed (even with this bit of code we can still not be assured that our code will be pre-JIT’ed as code pitching (a term I will save for a later post) could occur in between .. but it’s fairly likely)


The calling of this routine is pretty straight forward as shown in Figure 3. Basically we just define a delegate to the method we wish to test and pass it into our test routine.








    class Program{


 


        static void TestMethod() {


            System.Threading.Thread.Sleep(10);


        }


        static void Main(string[] args){


            TestHarness.Test(“Our First Test Function”, 100, new TestHandler(TestMethod));


        }


    }


Figure 3: Calling the TestHarness


 


One thing that is important to keep in mind with this test is that we should be testing fairly heavy methods. As an example we would not want to test XORing a variable against itself as opposed to setting it to 0 as our harness is simply not good at picking up such differences due to its relatively low resolution, in order to do such a test, our test method might do the operation 10,000 to 1,000,000 times. This methodology is also more reliable than simply running the code a single time as we are averaging our results.


Since many of the items we will be testing would use 100% CPU given the chance we must be careful about things such as context switches so that windows media player can keep playing Wally McClure’s ASP.NET podcast while we test. We cannot always safely assume that over long runs of tests that both sides will be equally affected by the context switching so we should always try to remove as many as possible while testing (yes this means turning the dancing girl in windows media digital enhancements off).


While I am on the subject of these types of tests, there are many people who suggest you build these types of tests into your unit tests. I cannot disagree enough with this as it will cause you many problems in the long run.


1)      Performance tests generally take a while to run (generally they are things like I need to run 50,000 iterations of this method in one second). These tests will raise the time in your code/build/test cycle.


2)      Although it is contractually important to express a requirement of running a method 50,000 times a second, the requirements do not state that it should be able to run 50,000 iterations in an unknown environment. If the machine running the tests is not up to snuff or is under a heavy load these tests will fail. My unit tests telling me that my machine is slow while I convert a .AVI to a DVD doesn’t really tell me much about my software.


I generally recommend the automating of such test but tend to put them in their own category and do not run them with my general unit tests (very similar to handling integration tests).


 



Hopefully this and the last post have set a solid foundation, in my next post we will use these two posts to actually have some fun digging deep into some JIT behaviors.


 


This entry was posted in Under The Covers. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

One Response to Performance Measurement

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>