Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

On UI Testing

I’m part of a secret society of developers that has evolved over the years into something that has had a pretty significant impact on my career. We ask each other advice, hang out at conferences, discuss trends in the field, etc, etc, and so on and so forth. Pretention is low, content and entertainment value is high. It’s essentially everything I had hoped alt.NET would have been.

A short while ago, we had a chat. It was the latest in a series, depending on how you define “series”, where we gather together to discuss some topic, be it JavaScript frameworks, OO practices, or smoked meat. On this particular day, it was UI testing.

I don’t recall all the participants but it was a good number of the people on this list. Here, I’m going to attempt to summarize the salient points but given my memory, it’ll more likely be a dissertation of my own thoughts. Which is just as well as I recall doing more talking than I should have.

Should you UI test?

This was a common thread throughout. Anyone who has done a significant amount of UI testing has asked a variant of this question. Usually in the form, “Why the &*%$ am I doing this?”

Let it not be said that UI testing is a “set it and forget it” affair. Computers are finicky things, UI’s seemingly more so. Sometimes things can take just that one extra second to render and all of a sudden your test starts acting out a Woody Allen scene: Where’s the button? There’s supposed to be a button. YOU TOLD ME THERE WOULD BE A BUTTON!!!

Eventually, we more or less agreed that they are probably worth the pain. From my own experience, working on a small team with no QA department, they saved us on several occasions. Yes, there are the obvious cases where they catch a potential bug. But there was also a time when we had to re-write a large section of functionality with no change to the UI. I felt really good about having the tests then.

One counter-argument was whether you could just have a comprehensive suite of integration tests. But there’s something to be said for having a test that:

  1. Searches for a product
  2. Adds it to the shopping cart
  3. Browses more products
  4. Checks out
  5. Goes to PayPal and pays
  6. Verifies that you got an email

This kind of integration test is hard to do, especially when you want to verify all the little UI things in between, like whether a success message showed up or whether the number of items in the shopping cart incremented by 1.

We also had the opposite debate: If you have a comprehensive suite of UI tests and are practicing BDD, do you still need TDD and unit tests? That was an interesting side discussion that warrants a separate post.


…is ongoing. There’s no getting around that. No matter how bullet-proof you make your tests, the real world will always get in the way. Especially if you integrate with third-party services (<cough>PayPal<cough>). If you plan to introduce UI tests, know that your tests will be needy at times. They’ll fail for reasons unknown for several consecutive runs, then mysteriously pass again. They’ll fail only at certain times of the day, when Daylight Savings Time kicks in, or only on days when Taylor Swift is playing an outdoor venue in the western hemisphere. There will be no rhyme or reason to the failures and you will never, ever be able to reproduce them locally.

You’ll add sleep calls out of frustration and check in with only a vague hope that it will work. Your pull requests will be riddled with variations of “I swear I wouldn’t normally do this” and “I HAVE NO IDEA WHAT’S GOING ON”. You’ll replace elegant CSS selectors with XPath so grotesque that Alan Turing will rise from his grave only to have his rotting eyeballs burst into flames at the sight of it.

This doesn’t really jibe with the “probably worth it” statement earlier. It depends on how often you have to revisit them and how much effort goes into it. From my experience, early on the answer is: often and a lot. As you learn the tricks, it dwindles significantly.

One of those tricks is the PageObject pattern. There was universal agreement that it is required when dealing with UI tests. I’ll admit I hadn’t heard of the pattern before the discussion but at the risk of sounding condescending, it sounds more like common sense than an actual pattern. It’s something that, even if you don’t implement it right away, you’ll move toward naturally as you work with your UI tests.

Data setup

…is hard, too. At least in the .NET world. Tools like Tarantino can help by creating scripts to prime and tear down a database. You can also create an endpoint (on a web app) that will clear and reset your database with known data.

The issue with these approaches is that the “known” data has to actually be known when you’re writing your tests. If you change anything in it, Odin knows what ramifications that will have.

You can mitigate this a little depending on your technology. If you use SpecFlow, then you may have direct access to the code necessary to prime your database. Otherwise, maybe you can create a utility or API endpoints that allow you to populate your data in a more transparent manner. This is the sort of thing that a ReST endpoint can probably do pretty well.


Consensus for UI testing on mobile devices is that it sucks more than that time after the family dinner when our cousin, Toothless Maggie, cornered—…umm… we’ll leave it at: it’s pretty bad…

We would love to be proven wrong but to our collective knowledge, there are no decent ways to test a mobile UI in an automated fashion. From what I gather, ain’t no picnic doing it in a manual fashion. Emulators are laughably bad. And there are more than a few different types and versions of mobile device so you have to use these laughably bad options about a dozen different ways.


What about companies that will run through all your test scripts on multiple browsers and multiple devices? You could save some development pain that way. But I personally wouldn’t feel comfortable unless the test scripts were extremely prescriptive. And if you’re going to that length, you could argue that it’s not a large effort to take those prescriptive steps and automate them.

That said, you might get some quick bang for your buck going this route. I’ve talked to a couple of them and they are always eager to help you. Some of them will even record their test sessions which I would consider a must-have if you decide to use a company for this.


I ain’t gonna lie. I like Cucumber and Capybara. I’ve tried SpecFlow and it’s probably as good as you can get in C#, which is decent enough. But it’s hard to beat fill_in ‘Email’, :with => ‘hill@billy.edu’ for conciseness and readability. That said, do not underestimate the effort it takes to introduce Ruby to a .NET shop. There is a certain discipline required to maintain your tests and if everyone is scared to dive into your rakefile, you’re already mixing stripes with plaid.

We also discussed Canopy and there was a general appreciation for how it looks though Amir is the only one who has actually used it. Seems to balance the readability of Capybara with the “it’s still .NET” aspect of companies that fear anything non-Microsoft. It’ll be high on my list of things to try the next time I’m given the option.

Of course, there’s Selenium both the IDE and the driver. We mentioned it mostly because you’re supposed to.

Some version of Visual Studio also provided support for UI tests, both recorded and coded. The CodedUI tests are supposed to have a pretty nice fluent interface and we generally agreed that coded tests are the way to go instead of recorded ones (as if that were ever in doubt).

Ed. note: Shout out to Protractor as well. We didn’t discuss it directly but as Dave Paquette pointed out later, it helps avoid random Sleep calls in your tests because it knows how to wait until binding is done. Downside is that it’s specific to Angular.

Also: jasmine and PhantomJS got passing mentions, both favorable.

Continuous Integration

This is about as close as we got to disagreement. There was a claim that UI tests shouldn’t be included in CI due to the length of time it takes to run them. Or if they are included, run them on a schedule (i.e. once a night) rather than on every “check in” (by which we mean, every feature).

To me, this is a question of money. If you have a single server and a single build agent, this is probably a valid argument. But if you want to get full value from your UI tests, get a second agent (preferably more) and run only the UI tests on it. If it’s not interfering with your main build, it can run as often as you like. Yes, you may not get the feedback right away but you get it sooner than if you run the UI tests on a schedule.

The main takeaway we drew from the discussion, which you may have gleaned from this summary, is: damn, we should have recorded this. That’s a mistake we hope to rectify for future discussions.

This entry was posted in UI Testing. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.Marisic.Net/ dotnetchris
  • http://www.davepaquette.com David Paquette

    Kyle, that is the approach I used with SpecFlow: Stating the data for the test as part of the Given steps. The bindings for those steps were able to setup the data using the data access layer from the app (in this case nHibernate repositories). I thought that approach worked well for us.

    The app also required a lot of seed/reference data for the database. To avoid thousands of lines of “Given” steps, we recreated the database into a known state once before running the whole test suite. For this we used a database backup.

  • http://www.Marisic.Net/ dotnetchris

    On this topic of testing and BDD i found that both! always to be the best

    xBehave (aka cukes) behavior tests with the gherkin files are great for the full stack tests that robo drive the UI. In a perfect world the gherkin language tests would actually be written by the business. Not very realistic but the business sure understands them when you discuss them together. I used SpecFlow for this in .NET with good success coupled with Selenium WebDriver and some common base steps we used for database setup/teardown.

    xSpecification (aka rSpec) code tests for test first development. I found Machine.Specifications (MSpec) with Machine.Fakes to be an amazing story for this in .NET along with its complementary test runner for Jetbrains Resharper. I only use these tests when i need to have mocked dependencies.

    If i have zero dependencies such as testing extension methods or algorithms i’ll likely just directly use xUnit/nUnit.

    Bonus shout out to Shouldly for assertions. Shouldly is amazing and you should use it in everything.
    Alright maybe it doesn’t make sense in assertions based on IO from WebDriver , but everything else! https://github.com/shouldly/shouldly

  • http://kyle.baley.org Kyle Baley

    There is a benefit in having your test explicitly describe the data it needs. E.g.

    Given I have a customer:
    | Name | Jack Skellington |
    | City | Halloweentown |
    And I have a product:
    | Name | Magic Sled |
    | Price | $30.00 |

    Reading this, you can explicitly see what data you’re working with directly in the test. Compare this to: Given I load up some test data that I’m pretty sure no one else has touched.

  • http://kyle.baley.org Kyle Baley

    Thanks for this, Dave. Lots of great information.

    The team where I did introduce Ruby was mostly successful. One person remained apprehensive about it but was no less productive than the rest of the team. At one point we considered migrating to JBehave and it was soundly defeated in a vote. But it was a pretty open-minded team and we had long-term goals.

  • Simon Timms

    This is is a great comment. I am curious about the work load behind creating a REST API or, in fact, any API for populating data for the tests. Do you not find that this is a lot of work to create and maintain instead of just writing a desired test state directly to the database? I can see the advantages to it as well in that it allows you to exercise some business logic as part of the population giving a closer representation of the real world.

  • Dave Haeffner

    Thanks for all this great info Kyle!

    Here are some thoughts that came to mind as I read your post.

    – I agree that automated UI testing is hard, and the same goes for data setup. But the pay-off can be there.

    – The cornerstone of successful UI automation is good, semantic locators in the DOM that are unlikely to change. Without them you have a lot of work ahead of you.

    – I’m a big fan of performing a Pareto analysis of existing features in an application to find the 20% of flows that are relevant (e.g., make the business money, are what users are using, have broken before, etc.) and focus automation on that. Only after getting these to run reliably will I move on to additional UI tests. I’ve written about this a bit more here: http://elementalselenium.com/tips/18-what-to-test

    – When possible, I perform data setup through a REST API. If that’s not possible, then through the database. And worst (absolute worst) case I write an automated UI script that performs setup/teardown actions through an admin interface.

    – Introducing Ruby to a non-Ruby shop is a challenge, but it’s not insurmountable. I’ve successfully introduced Ruby to 2 serious Java shops with good success. It definitely takes some doing though.

    – Mobile testing is still a Wild West (e.g., how desktop browser automation was 6 or 7 years ago). My hope is that the default solution will be something built on top of the WebDriver Wire Protocol (like Selenium is). Right now, the leading solution in that regard is Appium. Although I’m not convinced it’s going to win. Time will tell.

    – You can certainly get a lot of mileage with your automated UI tests (both for mobile and desktop browsers) when running them on a third-party service like Sauce Labs (https://saucelabs.com/). For mobile they offer simulators and emulators, and are starting to roll out real device support as well.

    – The Page Object pattern is table stakes for successful automated UI testing. Same goes for intelligent waiting. You mentioned Protractor’s ability to wait for binding to complete. Something similar can be accomplished within Selenium by using an explicit wait for targeted actions. I’ve written about this a bit at http://elementalselenium.com/tips/23-dynamic-pages and http://elementalselenium.com/tips/47-waiting

    – I agree Capybara is cool and has a helpful domain specific language (DSL). It’s a nice facade to place on top of Selenium too. And it can easily be integrated in a BDD tool like Cucumber. It’s great when it works, but it can be a real challenge to debug/fix when there’s an issue with the driver you’re using (e.g., Selenium). Over the years I’ve tended to shy away from tools like Capybara in favor of Page Objects and writing my own facade/DSL.

    – I also like BDD, but I see a lot of misuse and as a result have hesitations with it. This blog post from Liz Keogh articulates this really well and gives some helpful guidance on how to address these concerns: http://lizkeogh.com/2011/03/04/step-away-from-the-tools/

    – One thing you didn’t discuss which is relevant, and really cool, is automated visual testing. I’ve spent the last 9 months learning about it and using it. The gist: the future is here and for a few lines of code you can make your automated UI tests way way better (and automate something that was once thought to be only covered through manual testing — e.g., look and feel, responsive, etc.). And with a solution like Applitools Eyes (https://applitools.com/) you can address a lot of the false positives that can make automated visual testing inherently flaky. I’ve written quite a lot on the subject as it relates to open source and Selenium. Here are the write-ups I’ve done:


    Dave H

  • http://kyle.baley.org Kyle Baley

    That was one of the arguments that came up. The counter-argument was: it’s a pretty big “if”.

  • http://www.freekpaans.nl Freek Paans

    Nice post. I’m interested in the do you still need TDD/unit test discussion. Personally I think if you can make them really reliable and as fast as unit tests, you probably don’t.

  • http://kyle.baley.org Kyle Baley

    There’s always an ongoing inner battle on whether it’s worth it, especially at the beginning where you’re mostly fighting the technology (either your chosen UI framework, the web driver, or the browser itself). Because of this, UI tests have a way of exaggerating the good and bad parts of your team culture. It just takes one Debbie Downer, incessantly muttering about ongoing test maintenance, to add to your second guessing about their value.

    On a project where we leaned on them heavily, we benefited a great deal by bringing in an expert (www.orthocoders.com if you’re interested) to help set up the initial framework and train us on the technology and the techniques. Without this, there’s a decent chance we may have abandoned the exercise.

  • Chris Surfleet

    Generally agree with all this. We’re using CodedUI tests written in C#. They are great when you figure out that you have to ignore all the stuff about test recording etc and just hand-code everything. That is to the extent that we hand-code our map files which you’re not supposed to touch, its just easier that way!

    I think the single greatest help though is using common language – get your acceptance criteria written in cucumber, then create your PageObject layer (thanks, for that, we are using it but without knowing it was a thing) using the same language. Done right your tests can be very similar to your acceptance criteria, at least to the point that you can be ‘code reviewed’ by your manual tester!

    I’m still torn on weather it was a useful exercise. We’ve done it on smaller projects and its worked OK. but we haven’t had that ‘WOW that just saved our asses’ moment yet. Now we have it figured its probably time to try it out on a larger, more frequently modified project I suppose.

  • Pingback: The Morning Brew - Chris Alcock » The Morning Brew #1894