A reason to UI test, or “How to fill in the gaps”

Dja miss me? Of course not. You’re reading this in an RSS reader and if I hadn’t asked the question, you wouldn’t have noticed I was gone. If you’re like me, you’re scanning through until you can find any reason to skip this post and move on to the next one. Here, I’ll help your little mental filter:

Executive summary: An example of when UI/integration tests succeed where unit tests don’t.

But I’m warning ya, you’ll miss out on the amusing delivery if you go by this alone.


We are tantalizingly close to coming out of trial mode with BookedIN. My business partner had a nice little debate for our development process after its out. His stance: we need to write UI tests before we write code to avoid a lot of the issues that were arising. My stance: We unit test the ever-living raccoons out of the app now (thanks in no small part to Jukito). And we do have a UI testing process that was used for major features. While I agreed that we should have more, I was leery that it would make us any more productive. And I doubted it would help us discover any more issues than we already were…

(Ed. note: Did you notice the ellipsis at the end of that last paragraph? That’s what we in the legitimate journalism field call foreshadowing. Or, as is more often the case in my writing, foreboding. Note that you won’t see classy tropes like this from those hacks at Los Techies.)*

The bug

When someone logs into the application, there are a number of notifications that could pop up in various scenarios:

  • Owner logging in for the first time: Enter company details, following by a welcome message
  • New staff member: welcome message
  • Owner after the first time: You haven’t verified your email address yet (assuming it hasn’t, in fact, been verified)
  • Anyone after the trial period has expired: Message that says “Just kidding, you don’t have to pay! Tee hee to those that did.”
  • Existing users after a major release: What’s new!

There might be more, you can’t expect me to keep all this &*^% in my head.BookedIN

We covered all of this with a NotificationController. It would determine what conditions were met and start piling messages into a queue and display them in order. All of it very thoroughly tested with unit tests to make sure repositories were called and messages sent and users updated and so on and so forth.

The bug as listed was: When they log out and re-log in, users are seeing the what’s new message again.

As with most bugs, this was an oversimplication. As I tested, I discovered that this would happen intermittently. Sometimes it would happen as stated. Sometimes a new owner would enter her company details, log out, log in again, then see the company details window once more. Sometimes it would just work.

The analysis

“How can this be?” says I. The tests clearly outline what is supposed to happen and the code clearly passes all the tests.

As mentioned, we have a UI testing framework in place and used it on occasion. The reason it is not used more frequently is a topic that is already half-written but the short answer is: Google Web Toolkit adds some hurdles we haven’t had time to overcome yet. But given our recent debate, I fired up Notepad and banged out a capybara scenario. Nine times out of ten, it would fail. Except when I ran through the debugger in which case ten times out of ten, it would succeed.

The culprit

A good, old-fashioned race condition. In our client piece, the code for the first notification scenario above reads like this (it’s in pseudo-code because I don’t actually code anymore…):

if (logging_in_for_the_first_time) {
    showWelcomeMessage( );
    markWhatsNewMessageAsSeen( );

Judicious use of logging revealed the following exchange on the server:

[MarkingWelcomeMessageSeen] Starting execution
[MarkingWelcomeMessageSeen] Retrieving user details
[MarkingWelcomeMessageSeen] Marking welcome message seen
[MarkingWhatsNewMessageSeen] Starting execution
[MarkingWhatsNewMessageSeen] Retrieving user details
[MarkingWhatsNewMessageSeen] Marking what’s new message seen
[MarkingWelcomeMessageSeen] Saving user
[MarkingWhatsNewMessageSeen] Saving user

After making a mental note to perhaps differentiate my method names a little better, it was painfully obvious what the problem was.

The solution

That’s not the topic of this post. Focus, Hillbilly!

The moral

As I said, all unit tests passed. Our tests on the client proved that we were marking the messages seen in the appropriate conditions. Tests for the server component show that we are calling repositories and setting properties (or the Java equivalent: setting properties, but with more code) properly.

But the problem didn’t manifest itself until we did the UI test (or integration test, if you prefer).

So folks, heed my warning. A lack of decent integration/UI tests can lead to a smug and self-righteous business partner.

Kyle the Humbled

* shiny nickel for those of you who recognize this little bit being ripped off from Bloom County

This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://twitter.com/zkhan Zubair Khan

    How good would a UI test that failed once every five runs be either? I think what the moral of the story here actually is, “We’re never gong to get rid of exploratory testing, dammit”.
    You find this kind of bug, then you unit test the fix for it, to make sure it’s not inadvertently undone later.

  • http://www.surgeforward.com SaaS

    Good post, thanks for the humor it always makes reading fun. I want to know what the fix is though. I have had som problems with this same thing and though I can almost see it my minds eye, the problem, as a whole, eludes me. I would be interested in reading about it.

  • http://twitter.com/chr_horsdal Christian Horsdal

    Thanks for clarifying. It makes perfect sense to avoid/guessing what the framework does. That would just lead to fragile tests.

  • http://kyle.baley.org Kyle Baley

    Christian: This is a GWT application and the race condition was being triggered by two successive JavaScript calls. We’re also running on Google AppEngine so in order to run an integration test, we’d have to figure out what GWT was doing behind the scenes to trigger the calls.

    All this boils down to: it’s easier to write UI tests than subcutaneous integration tests, especially since we’re using Capybara. The test to verify it is:
    Given I started registration as “Merle Haggard”
    When I go to “the verification uri”
    And I log in
    And I follow “Sign Out”
    And I log in as “merle”
    Then I should not see “What’s New!”

    While UI tests take longer to run, there’s a great deal of comfort to having a test that goes through the exact set of steps used to reproduce a bug in order to verify that it is fixed.

    That said, we do have a number of integration tests and I’m a firm believer of them. They just don’t go entirely end-to-end.

  • Phill

    Off topic but can someone look at the RSS feeds, they are sending duplicates now.

  • http://twitter.com/chr_horsdal Christian Horsdal

    But the example isn’t really UI per se. Its just client side logic, so an integration test will do. No need to go through the UI, is there? What am I missing here?

  • Pingback: Tweets that mention A reason to UI test, or “How to fill in the gaps”: Dja miss me? Of course not. You’re reading this in an RSS rea... -- Topsy.com