How to make your build faster

Screen Shot 2012 03 26 at 5 47 40 PM
I did an informal survey a while ago and the results were frankly abysmal. I cannot understand how people work in such environments. I am ADD, if I have a feedback cycle more than a few seconds I have already hit ctrl+right arrow and am reading my email or sitting in twitter. This however has a very serious affect on my work. I am constantly pushing and popping my mental stack of what was going on (more often than not having cache misses as well). The sample size was pretty decent as well with over 700 respondents.

This post is going to look at some ways we can make our build times faster. There are loads of ways aside from buying SSDs. I work in a VM on a non-SSD drive on a laptop and rarely see a build over 5 seconds mostly because I focus very heavily on not having solutions with 20,000 projects in them but there are some other ways as well.

Project Structures

To start with we can use smarter project layouts. Patrick Smacchia has some good posts on it:

Advices on partitioning code through .NET assemblies

Hints on how to componentize existing code

Keeping well factored projects can go a long way and making a larger project will be faster than building many smaller ones (why? as an example they have a tendency of reusing many of the same references etc that need to be loaded repeatedly causing far more i/o).

Parallel Builds

These are good reading to start but there are many more options that are available. Let’s start with one of the comments from Ben “I think unless you go down the parallel build/test route (like NCrunch) then this issue of build times is not going to go away.”

What a great selling point. Luckily for builds everyone can already do this. Did you know that Visual Studio and MSBuild already do parallel builds? All you have to do is drop into your configuration and you will see an options screen like the one here

Screen Shot 2012 03 26 at 4 39 49 PM

Put in your max number of cores that you want. Work with msbuild? just use /maxcpucount:4. Of course this is still rather limiting as if you have project 1 that references project 2 which references project 3 you are in a serial situation anyways. The maxcpucount represents the number of projects that can be built concurrently (e.g.: providing no dependencies). Both MM and nCrunch support this option as well. This can get your build times to be quicker in some scenarios though I tend to see only a +- 20-25% increase on average.

Mighty Moose

We did a quick survey on Mighty Moose users about a week ago and I was amazed to see that very few people were using this feature (about 10% of a survey size +- 100-150 people). Mighty Moose has a smart build system inside of it that is actually much better than what msbuild can do because it has more information. I know dotnetdemon from red gate has something similar to it.

Basically since I know what is changing and I know how that affects the dependency graph I can be much more intelligent with my builds. I also have further knowledge through static analysis about what the affects of that change was (are public contracts changed etc?). Of course its a bit more complicated than this in practice (lots of edge cases) but it can make a huge difference in your build times for your typical builds in your TDD cycle.

[youtube: 05Z5sPrGEeo]

You can see the difference in the video above. MSBuild on an incremental build in AutoTest.Net about 4.5 seconds. MM for the same build + locate tests to run + run the tests about 2.5-3 seconds. AutoTest.Net itself is not a tiny piece of code either. It has a reasonable amount of code in it, even the incremental build is doing pretty well when you consider my dev machine is a VM on a non-SSD drive. This is because we have more information than msbuild does as we stay running compared to it running then dying. You can enable the setting on the first screen in config (Build Setup) here.

Screen Shot 2012 03 26 at 6 17 23 PM

Some people have asked if its so much faster why do we make you opt in to using it? There are a few reasons for this. The largest is compatibility mode. Most people do not generally build their projects directly and often have reasons why they don’t actually build successfully. Building the solution gives the highest level of compatibility with existing VS projects. As such we make you opt in for it, but its worth opting in for!

This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post.

10 Responses to How to make your build faster

  1. gregyoung says:

    @kieryn you are rather obviously not doing TDD then. If you were doing TDD your cycles would be very painful at 20 seconds each (on a good day I write between 100-150 tests). Thats a minimum of 200-300 builds for test runs.

  2. gregyoung says:

    @anonymous I would be happy to show that (I can run the second build 5 times and its +1 second difference). The reason for this is that they build different things. The MM build system there ends up only building 1 project where as MSBuild in incremental mode does 5. Would you prefer to see msbuild on command line or in visual studio :)

  3. gregyoung says:

    It runs my tests every time I have a build pass (only the tests needed for the change). My build runs every time I save. You can also configure to only do it when you build manually if you want.

  4. gregyoung says:

    I would be happy to post that. The reason its faster is we are not building 4 of the projects that the msbuild incremental build builds.

  5. Anonymous says:

    In general, try your speed comparison in the other direction too (the fast one first) – most likely they’ll be very similar to the original results but sometimes vary dramatically due to HD caching and such. HD read caching affecting just one of multiple similar actions has skewed many such comparisons into erroneous conclusions before.

  6. Trevor says:

    From a C/C++ perspective its a common error to have a global includes file that lists all the other .h files in a project, so one change on any .h file forces a rebuild of the entire code set.

  7. kieryn says:

    I use ReSharper to provide instant feedback on build errors. My current solution is about 16 projects with a build time of about 20 seconds. However, I get near-instant feedback from ReSharper about solution errors. This vastly reduces the number of times I need to build. I build and run only when I have completed a feature in order to test that feature, and with ReSharper reporting no errors I have very high confidence that the solution will just build and run successfully.

  8. Rob says:

    I’ve spent many moons optimising build speeds usually to try and get them under one minute. Asynchronous build tasks, multi-core builds (as you mention), centralising build output (so dll’s aren’t copied all over the place), cutting down project counts.

    Luckily these days SSD’s take care of all of this for me. Aside from havig one, you just have to make sure any steps you take to speed up the build aren’t papering over the cracks. A slow build is a message that your solution may be overloaded (time to split the app?) or your unit tests aren’t very well written etc… Paper over the cracks by all means, just make sure you address the root cause too.

  9. Don says:

    And you want to run your tests _every time_ you hit ctrl+s !? Btw, you don’t need 20,000 projects, but a hundred is enough to take build time to the minutes.

    The video doesn’t convince me, that looks like a toy project. What should that prove!? That they named the button “Update” instead of “OK” is driving me crazy enough already. And where’s the cancel button? Can’t people follow simple UI standards?

  10. James Kovacs says:

    Good advice. One note on parallel builds… If you pass /maxcpucount (or /m for short) to msbuild without an argument, msbuild will default to using the number of cores available.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>