Mock Framework Benchmarks

UPDATE: I tracked down the issue and committed a patch to Rhino.Mocks. Rhino.Mocks is now much more competetive performance wise, our CI build time nearly halved, and about 4 minutes out of 7 of our test time has disappeared. New numbers below.

I’ve complained before that Mocking is Slow but I never really dove further into it. Today I decided to actually compare Rhino.Mocks to other mock frameworks on a pure performance basis to see if it was a global problem. I timed 2000 unit tests across 100 classes with 20 tests each. The results were a bit surprising:

Framework TD.NET Time nunit-console Time
Rhino.Mocks old trunk 57.36s 28.82s
Rhino.Mocks new trunk 22.94s 7.59s
Moq trunk 18.30s 5.91s
TypeMock 4.2.3 Reflective Mocks 15.36s 9.35s
TypeMock 4.2.3 Natural Mocks 16.92s 9.56s

That’s right, according to these tests, Rhino.Mocks is at least 3 times slower than the other frameworks when under heavy load in TD.NET and five times slower in the console according to these tests. It’s also interesting to note that TypeMock is faster than Moq in TD.NET, but slower in the console runner.

While running the Rhino.Mocks tests it is very clear that there is a degrading performance issue. All the other frameworks executed tests with a near constant speed per test, but Rhino.Mocks slowed down noticeably about half way through.

Please feel free to try it yourself, grab the project here. You should be able to just run the 4 strategy .bat files (run-rhino, run-moq, run-tmock-reflective, run-tmock-natural). Let me know if you find anything interesting.

This entry was posted in frameworks, mocking. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://codebetter.com/members/aaronjensen/default.aspx aaronjensen

    Hi Royo,

    No offense, but I don’t know who you are or what release/mocking framework you’re talking about. Feel free to rerun them yourself and blog about it though or send me the results. The code is in the article.

  • Royo

    time to re run these.
    we made major speed improvements in the latest releases.

  • http://www.jameskovacs.com james.kovacs

    I was equally confused when looking at a performance trace of Moq vs. Rhino. (I don’t have TypeMock installed.) The majority of the time in both Moq and Rhino was spent creating proxies and both frameworks are using DynamicProxy2 for proxy creation. Something was definitely up. Excellent work tracking down the problem!

  • Aaron Jensen

    I don’t really consider NMock2 a framework worth considering, as it lacks a strongly typed, refactor friendly syntax, so no, I didn’t. Feel free to add a Strategy though and give it a shot.

  • http://lextm.cn Lex Y. Li

    What about NMock 2? Did you test on that?

  • http://colinjack.blogspot.com Colin Jack

    Does this contribute to the 2 articles you have to write about liking TypeMock for the free license? :)