Aaron Jensen

Sponsors

The Lounge

Syndication

Recent Posts

Tags

View more

Archives

Advertisement

Aaron Jensen » Mock Framework Benchmarks
Images in this post missing? We recently lost them in a site migration. We're working to restore these as you read this. Should you need an image in an emergency, please contact us at imagehelp@codebetter.com
Mock Framework Benchmarks
UPDATE: I tracked down the issue and committed a patch to Rhino.Mocks. Rhino.Mocks is now much more competetive performance wise, our CI build time nearly halved, and about 4 minutes out of 7 of our test time has disappeared. New numbers below.

I've complained before that Mocking is Slow but I never really dove further into it. Today I decided to actually compare Rhino.Mocks to other mock frameworks on a pure performance basis to see if it was a global problem. I timed 2000 unit tests across 100 classes with 20 tests each. The results were a bit surprising:

Framework TD.NET Time nunit-console Time
Rhino.Mocks old trunk 57.36s 28.82s
Rhino.Mocks new trunk 22.94s 7.59s
Moq trunk 18.30s 5.91s
TypeMock 4.2.3 Reflective Mocks 15.36s 9.35s
TypeMock 4.2.3 Natural Mocks 16.92s 9.56s

That's right, according to these tests, Rhino.Mocks is at least 3 times slower than the other frameworks when under heavy load in TD.NET and five times slower in the console according to these tests. It's also interesting to note that TypeMock is faster than Moq in TD.NET, but slower in the console runner.

While running the Rhino.Mocks tests it is very clear that there is a degrading performance issue. All the other frameworks executed tests with a near constant speed per test, but Rhino.Mocks slowed down noticeably about half way through.

Please feel free to try it yourself, grab the project here. You should be able to just run the 4 strategy .bat files (run-rhino, run-moq, run-tmock-reflective, run-tmock-natural). Let me know if you find anything interesting.

Posted Thu, May 8 2008 8:50 AM by aaronjensen
Filed under: ,

[Advertisement]

Comments

Colin Jack wrote re: Mock Framework Benchmarks
on Thu, May 8 2008 1:26 PM

Does this contribute to the 2 articles you have to write about liking TypeMock for the free license? :)

Lex Y. Li wrote re: Mock Framework Benchmarks
on Thu, May 8 2008 10:24 PM

What about NMock 2? Did you test on that?

aaronjensen wrote re: Mock Framework Benchmarks
on Fri, May 9 2008 12:25 AM

I don't really consider NMock2 a framework worth considering, as it lacks a strongly typed, refactor friendly syntax, so no, I didn't. Feel free to add a Strategy though and give it a shot.

Aaron Jensen wrote Rhino.Mocks Performance Issue Fixed
on Fri, May 9 2008 1:56 AM

Just wanted to quickly note that I tracked down the performance issue in Rhino.Mocks and patched it

Community Blogs wrote Rhino.Mocks Performance Issue Fixed
on Fri, May 9 2008 2:29 AM

Just wanted to quickly note that I tracked down the performance issue in Rhino.Mocks and patched it

Reflective Perspective - Chris Alcock » The Morning Brew #90 wrote Reflective Perspective - Chris Alcock » The Morning Brew #90
on Fri, May 9 2008 3:04 AM

Pingback from  Reflective Perspective - Chris Alcock  » The Morning Brew #90

Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew wrote Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew
on Fri, May 9 2008 8:47 AM

Pingback from  Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew

Code Monkey Labs wrote Weekly Web Nuggets #11
on Fri, May 9 2008 10:57 AM

Understanding Model-View-Controller : Introducing developers to the MVC concept can, at times, be a challenging endeavor. Jeff Atwood has put together a very informative post that may be able to help! Mocking Frameworks Benchmarked : In search of a faster

james.kovacs wrote re: Mock Framework Benchmarks
on Sat, May 10 2008 2:09 PM

I was equally confused when looking at a performance trace of Moq vs. Rhino. (I don't have TypeMock installed.) The majority of the time in both Moq and Rhino was spent creating proxies and both frameworks are using DynamicProxy2 for proxy creation. Something was definitely up. Excellent work tracking down the problem!

StevenHarman.net wrote Rhino.Mocks Beta Bits are Running Turbo-Charged!
on Wed, Jun 11 2008 12:07 AM

Rhino.Mocks Beta Bits are Running Turbo-Charged!

Code Monkey Labs wrote Weekly Web Nuggets #11
on Sun, Feb 22 2009 10:50 PM

Understanding Model-View-Controller : Introducing developers to the MVC concept can, at times, be a challenging endeavor. Jeff Atwood has put together a very informative post that may be able to help! Mocking Frameworks Benchmarked : In search of a faster

Royo wrote re: Mock Framework Benchmarks
on Thu, Dec 10 2009 6:32 PM

time to re run these.

we made major speed improvements in the latest releases.

aaronjensen wrote re: Mock Framework Benchmarks
on Thu, Dec 10 2009 9:03 PM

Hi Royo,

No offense, but I don't know who you are or what release/mocking framework you're talking about.  Feel free to rerun them yourself and blog about it though or send me the results. The code is in the article.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?