Using a gmail account as a production bug tracking server

I’d like to share here an idea I implemented 2 years ago in NDepend and that has proven being pretty powerful and useful: Using a gmail account as a production bug tracking server !

By production bug tracking server, I mean a product that can log through the web information relative to bugs that happen on clients machines, during production execution.

By surfing the web, you’ll certainly find OSS or commercial production bug server products. One of the most popular product is certainly FogBugz. But whatever the product you’ll choose, it will necessarily come at the cost of investing time to understand how it works, to configure it, maybe to pay for it and to host it somehow.

Before investing resources in such product, I wanted to implement quickly this idea of using a gmail account as a production bug tracking server. After all gmail is an extremely powerful product, I already know well how to use it and, by imagining that a bug occurence = an email, it is actually adapted to host bugs tracking.

The only work I had to do was the burden of transforming an unhandled exception into an email! Basically the NDepend bug tracking gmail account looks like that:

Each email represents an unhandled exception, and the email subject is what I call the exception hash that identifies the bug through: the NDepend version, a 4bytes hashcode made from the exception stacktrace and the exception type. With this hashcode and the gmail search capabilities, I can readily know how many times a bug occured and its frequency.

For each unhanded exception, the bug tracking system logs all relevant information including: The exception handler context short description (did it happened in NDepend addin for VisualStudio 2010? while starting VisualNDepend.exe? while running an in-process Analysis?…), the .NET Fx version, the OS version, the processor architecture (x86, x64). But also the stack trace (obfuscated) and other information about the exception, including eventually the InnerException information, and all assemblies loaded in the current AppDomain with their versions. Actually I don’t log the OS/.NETFx/VisualStudio languages because this information is given in the Exception.Message and so far, we didn’t stumbled on many language specific bugs (although we had a few, especially because of Japanese installations).

The added value of using gmail, comes from its great capabilities to search in mails. It is really easy to identify if a bug occurs only in a certain context (like only on x64 machine or only in VS2010 when the VS2010 extension XYZ is loaded).

Another cool gmail capabilities is to create rules to do actions on incoming emails. It is pretty easy to create a rule to skip inbox for fixed bugs. In certain situation, it is also great to create some email tag like Bug XYZ reproduced. The bugs are of course identified from part of its content, like the exception hash described above.

With such bug tracking system, NDepend stability is now lower than 1 unhandled exception every 3.000 runs (the number of runs is logged differently through the licensing process). Of course we are not stupidly cheating and we let all exceptions bubble up.  This number is pretty cool if you take account that most remaining unhandled exceptions come from Visual Studio addin API (everyone that had to develop a VS addin understand what I am talking about!).

 

If you are currently using a bug tracking product, I’d be curious to hear about killer features you’d miss by using such a gmail system. I can imagine Integration with Source Control server or Stacktrace de-obfuscated automatically. What else?

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Cyber Sinh

    Hi Patrick,

    Could you share with us the code to compute the hashcode from exception?

    Thanks.

  • http://www.key-logger.ws/ Key logger

    GReat information, very useful for all us

  • Wilson Edgar

    Love the ideia, I’m actually implementing it on one my websites that has to invoke a few web services, and so far has been it’s been a very helpful.

  • http://devtalk.net Dmitri

    I just use FogBugz. Gmail is nice, but I like a real bug tracker, not just an email account that I have very limited control over. Also, I like my Hg sources being logically tied to the bug tracker, it makes it a lot easier to check who did what, when and in response to which request. Not to mention code reviews and such.

    Also, I’ve recently been looking at SmartAssembly and the way it packages up exceptions and mails them for you. Seems like a good fit with FogBugz.

  • http://twitter.com/moutasema moutasem al-awa

    I love the idea

  • Anonymous

    This is an awesome tip Sarah! Thanks   Really hope this lab setting will end up in the main settings

  • https://plus.google.com/107047796519061210494/posts Sarah Price

    Actually, we do have that setting. Go to your Mail settings > Labs > Signature Tweaks. Once you turn it on, you can put your signature above the quoted reply. Enjoy!

  • http://dariosolera.it Dario Solera

    Well, our application is a SaaS so I guess it’s simpler as we don’t even have the problem of keeping the SMTP account secret. We usually send error reports directly to our support email address (which is Google Apps), but tnless we work on the problem right away, the bug gets filed in a real bug tracker (Codebase).

  • Anonymous

    >but then we’d need to provide the account password when sending the email.

    We just log the exception data somewhere on the web, and then send the email from the server that receive the log.

    >And then there are the firewalls.

    Or even zero connectivity as for some developer machines. This is not a problem as long as this ratio remains constant. What we want is an holistic view of bugs that manifest in production, and these bugs happens with the same statistical distribution on connected and disconnect machines.

  • Peter Aglen

    We have played around with this idea, but have run into two main problems:

    What SMTP server to use? This simple answer seems to be Google’s own, but then we’d need to provide the account password when sending the email. How could we make sure that someone doesn’t dig out the password from our assemblies?

    And then there are the firewalls. Most of our customers are sitting behind some rather paranoid firewalls. They are basically not letting any traffic out unless it’s on port 80 or 8080. Any way around that?

  • Anonymous

    Thanks for the kind word on NDepend Erik :)

    >I’m not sure that it couldn’t be somehow extended via an elaborate email
    rule and exception reporting system to do just about anything.

    Indeed, while gmail is extremely powerful, not everything can be done. But it is my choice to add email tag, or even email bug description content, to make it work better with gmail capabilities.

    >Of course, now you have two systems to deal with and maintain.

    You get it! Working with 2 systems would hurt me. If one day the gmail bug racking system comes with too many limitations (which shouldn’t happen tomorrow, nor in 2012) I’d prefer  migrate to another one consistent bug tracking system.

  • Anonymous

    >I can see some disconnects between automatic bug submission and other
    bug submissions where a customer actually sends you an email directly
    with a screen shot of a UI bug for example.

    Not really because when an exception popups in NDepend, the user gets an error form with the exact same exception desc text, being asked to click a button to forward us this content with a description of what happened. Screenshot can indeed not being attached, but a reference to the user email in our support box is enough (so far).

    >I really like having everything in one place and integrated in with the IDE.

    Most of data (except pic) are logged in the code itself, hence in the IDE. With the big advantage of being logged closed to the suspicious culprit method.

    >Also, hashes, stack traces, etc. in code comments may be hindering
    because their meaning can get out-dated after the code is refactored for
    example.  I don’t think they belong in the code itself.  I think this
    also touches on best practices for code comments.

    The vast majority of identified bugs MUST be fixed before a new release. And if the code gets refactored, its primarily to fix the bug itself. But indeed, we are accumulating some not-reproductible bug  in comments in code, but for that to happen, this must be a very rare bug (occurs less than 1 every million run). This kind of one-shot bug typically comes from kinda corrupted Windows / .NET Fx /third-party software installation. And if the concerned code gets refactored, we have the choice to keep the bug desc comment or discard it if the refactoring is significant enough to eventually fix the bug.

    But Rami, this way of managing bugs works great on the small dev team of NDepend. But my opinion is necessarily biased since I invented and massively use this system. I guess that if you are used to other bug tracking standard, or if you are manging a team made of dozens of developers, using gmail as bug tracking system comes with too many limitations for you.

  • Anonymous

    Glad you enjoy it Sarah, and keep up the good work with gmail, it just rocks! Btw, I see you are community manager so I have one feedback: The main (and pretty only) tack I have with gmail is when responding an email, the automatic signature is put after the answered content. While it makes sense in a few situations, most of the time I’d like it to be put just before, this way I can write my quick answer and send it, without copy/pasting the signature. A tickbox in the signature setting would be welcomed.

  • Anonymous

    Actually the way it works now already harness the notion of ‘conversation’ thanks to the exception hash subject. But in these conditions gmail seems to bind in conversations only emails with same subject received in the same 24 hours.

  • https://plus.google.com/107047796519061210494/posts Sarah Price

    What a cool use of Gmail!

  • Rami AbuGhazaleh

    I can see some disconnects between automatic bug submission and other bug submissions where a customer actually sends you an email directly with a screen shot of a UI bug for example.  I really like having everything in one place and integrated in with the IDE.  In the latter case the screen shot would be added as an attachment to the bug work item for example.  Also, hashes, stack traces, etc. in code comments may be hindering because their meaning can get out-dated after the code is refactored for example.  I don’t think they belong in the code itself.  I think this also touches on best practices for code comments.

  • Beau Fabry

    Cool. Have you thought about only putting the last line of the stack trace in the subject and then the hash code in the body? That way you should get all of the same exceptions from different servers in a single ‘conversation’.

  • http://www.daedtech.com Erik

    First of all, I’ve been using NDepend for a year or so, and I find it indispensable.  My compliments.

    As for the use of gmail as a tool, I’d be intrigued to see a demonstration, and I’m not sure that it couldn’t be somehow extended via an elaborate email rule and exception reporting system to do just about anything.  However, since that isn’t really the primary purpose of gmail (at least not until some chrome plugin comes along :) ), I’m wondering if the use of gmail couldn’t become a step to streamlining bug tracking with established products.  Something like a process flow of:

    1.  Exception is generated, triggering an email to gmail account.
    2.  Some process monitors a gmail account and creates bug reports in bugzilla/clear quest/fogbugz/etc.
    3.  Whoever gets assigned the bug fills in any information not auto-generated.

    It seems like this could conceivably provide the best of all worlds — you keep the system you’re familiar with, you get the quick turnaround from generated exception to bug report, you retain the ability to search through gmail, etc.  

    Of course, now you have two systems to deal with and maintain.  Anyway, just a thought on one of the first things that occurred to me as I read this.

  • Anonymous

    What I didn’t precise in the post, and indeed is important, is that the bug info ( repro steps, resolution, notes, assigned to,) are stored in the code itself through comment, tagged with a set of tag like TODO_BUG… The exception hash is also part of the bug description in the code, so it is easy to trace back to the gmail account.

    These comment are added periodically, both from users bug feedback and from the gmail bug tracking system (that often gives essential additional info on users feedback).

    The advantage of having this bugs data in the code itself are:
    -> no need to have a third-party server to log bugs info
    -> no need to browse the code to know where the buggy code is. The buggy method (or at least the most suspect buggy method) is the one that contains the bug comment.
    -> the code is necessarily in-sync with the bug database

    But I agree that with this simple, yet efficient, technique we lake things like bug stats and history (some of the stats and history can be obtained from the Release Notes).

  • Dylan Smith

    Things I’d miss:

    Workflow (which I suppose can be accomplished by creating folders to represent workflow states).

    Adding other information to the bug, such as repro steps, resolution, notes, assigned to, etc

    Ability to link back to requirements or test cases to have tracability.