As some of you may know Svein (@ackenpacken) and I have decided to make Mighty Moose free. This is not a decision that is being take lightly and much thought has gone into it. This post is to explain why we went free and the future.
Almost two years ago now sitting in the Dubliner in Oslo we started having a discussion with Einar about AutoTest.Net. Wouldn’t it be cool if AutoTest.Net could only run the tests that needed to be run when a piece of code changed. We looked in the market and there was no software that could do it well (we only actually found one named infinitest in the java space and it was at the time (I have not tried it lately) very easy to trick).
We discussed with quite a few people the idea and were continually told that “its impossible”. We like “impossible problems”. After some quick spikes we even discussed it as being “non trivial but not hard” we figured a prototype could be working in about a month (given the usual software +-50% “fluff”). I love that now its no longer considered impossible but an expected behaviour of such tools
I returned to Montreal and started working on the bit that did the selection, it was dubbed the “minimizer” and worked off of static analysis (we realized very early on that code coverage is not a good solution to the problem of which tests to run as it describes the last run which may or may not have anything to do with the current run). After one month there was a prototype there, unfortunately it ran way too many tests. Often it would produce results of 200 tests when you really would expect 8 to run. After another 4-6 months of work it started to give pretty reasonable results in most situations. Sounds like a classical software estimation process to me
But enough reminiscing …
Mighty Moose is from this point forward free. Not “free in beta” or “free with a bunch of functionality turned off” but free (license is being updated as I write this). We announced this rather quietly at NDC.
Many people have asked me “why would you go free? you could charge for what you have.”. This is true. Continuous Testing tools will be taking off and yes we could charge for it however its not quite as simple as that.
To start with support is generally included with a paid product such as a dev tool. Let’s try some really quick math. Let’s say we were wildly successful the first year and got 2000 paying daily users (this would mean we would likely have at minimum double this actual users as you often give away free versions to user groups etc). This would suggest we have very deep penetration into the .NET TDD community (how many TDD’ers do you think there actually are?)
So 4000 users. What could we possibly charge for such a tool? USD$250 would seem to be about the average pricing our research suggested. So we would in the first year have a top line revenue of $500k USD. That sounds like a lot but lets start breaking that down further. For 4000 users how many dedicated support guys would be needed?
One interesting thing I have found is that very often a tool such as Mighty Moose (or any continuous testing tool) is the messenger of a problem not really the problem itself. This could be versioning issues with mSpec (I believe this is fixed now), never having run a redirected build before (this needs to be done as VS loves to lock files), or even just a new release of the CLR/Unit Testing Framework/etc. If you were to look at the width of support of a continuous testing tool its build automation + unit test running + profiler/analysis. This is very wide considering the price of the tool.
Along with this as dev tools are built to be very flexible quite often you will need to do some level of configuration on an existing project in order to get things working properly (out of box can only go so far .. even if it works you likely need to optimize things). It is not hard to trick any such tools. Just put a post build step that manipulates your output e.g.: moves files around (every such post build step would need custom integration written to understand it).
Even worse the support guys you have need to not only be developers but be good developers. In other words the role is basically 50% support 50% developing new features (dev team does support). You could try with just regular support guys and front line vs second line support but this will probably just annoy many users though it could possibly lower blended cost of support (e.g. front line + dev team). These methodologies however also annoys your developers as they are now doing direct support.
In order to get around the support cost question you are left with three possible options. The first is a no go from the start, support contracts. Developers don’t want to pay yearly fees on support costs. The second is to reach high levels of growth but an issue here is how many people actually write tests (let alone do TDD) in .NET. This brings the third model that most developer tool vendors use, I release a new major version of the software once per year.
This “major version release” strategy basically allows people to “upgrade” to the newest version. Generally I also will sunset support with this model to force upgrades (or at least lower support costs for older users). This model has been quite successful for many companies but comes with its own drawbacks. The largest drawback is that I as a company need to release a version of software whether or not I can come up with things that are really of value to my customers. Often I will end up with a few really valuable things but just listing those is not enough to get people to upgrade (think about a change log with 7 items on it…) so I start piling in crap to make a more impressive change log. This crap comes at a cost to the 80% of my users that don’t care about the particular feature and I end up building “mega tool”. This pattern can be seen over and over again whether we talk about Visual Studio itself, Microsoft Word, or any of the popular VS adding products.
This actually leads to a second issue as well. When in beta we have focused very heavily on features that only really interest the top 5% of developers (not a good base for a product, for a product we would be much more interested in the early/late majority areas of the s-curve). One question we get quite often is “how well will things work on my solution with 300 projects”. Now that we have said that we are free I can honestly tell you “I don’t care, when you have a solution with 300 projects in it you have bigger problems to worry about than getting continuous testing working”. Flashy new tools are not the answer, you should stop trying to hide your pain and start doing root cause analysis.
If we were to go down the lines of a paid product we would basically be forced to stop innovating. Our business goals would not be well aligned with our personal goals of changing the way people code. Our bread and butter would be the early/late majorities who want to use things in a much different way (they want slow incremental change) than the early adopters and on much different looking code bases. An example of this can be seen in a common feature request in our graphs of “making them look better and come up faster when there are more than 1000 nodes to show” (they come up with very small nodes). Yes we can do this (and spend 3 months of developer time working on it) but I am entering a point of alleviating pain instead of dealing with the underlying symptom (how on earth do you understand code that puts 1000 nodes in a graph, the graph will not solve this issue for you). This is also 3 months of developer time that we were not working on something actually useful.
I personally believe that the future is not in making tools that alleviate pain that many have but in helping them identify root causes of the pain and resolving it.