Automated Web Testing with Selenium Driven by .Net

The Agile development community has struggled for years with an array of solutions for automated testing solutions for web development.  NUnitASP is a good way to unit test server side ASP.Net code, especially now that it doesn’t require XHTML compliant pages, but it can’t handle client side scripting and AJAX is exploding in popularity.  Several tools have used COM (must die) to drive Internet Explorer (IE) with varying degrees of success.  My personal experience is that the IE COM API is too byzantine and flat out flaky.  Besides the flakiness, Firefox and other browsers are gaining in popularity so the IE only testing might not cut it anymore.

Enter the Selenium project.  The developers of Selenium had the brilliant, but in retrospect painfully obvious, idea to use Javascript inside a browser to drive the web testing.  Presto, automated testing for web applications that can test client side Javascript and multiple browser engines.  In its original incarnation Selenium ran FIT style test tables inside a web browser.  Recently, Selenium Remote Control has been released to that allow the core Javascript engine to be driven through API’s for .Net, Java, Ruby, or Python.  The FIT style table runner is perfectly functional, but for us it’s very convenient to use the Selenium RC .Net wrapper.  So far I’m pleasantly surprised by how easy it is to use the .Net wrapper.  

Sample Test

The API libraries communicate with the Selenium Server, a Java executable that can start and stop any supported browser and send testing commands to the browser.  The .Net callable wrapper works by sending HTTP messages to the Selenium Server that controls a browser.  The first step is to start up the Selenium Server from a command prompt.

java -jar server\selenium-server.jar -interactive

The next step was to create a simple HTML page with a textbox that changes values when a button is clicked:

<html>
<head>
<script language=javascript src=prototype-1.4.0.js></script>
<title>Selenium Target Page</title>
<script id=clientEventHandlersJS language=javascript>
<!--
 
function button1_onclick() {
    $('text1').value = "Goodbye"; // Using the Prototype library
}
 
//-->
</script>
</head>
<body>
 
<form name="form1">
    <input id="button1" type=button onclick="return button1_onclick()"
 value="click me"/>
    <input id="text1" type=text value="Hello"/>
    <select testid="select1" >
        <option value="1">North</option>
        <option value="2" selected=true>West</option>
        <option value="3">South</option>
        <option value="4">East</option>
    </select>
</form>
 
</body>
</html>

Next I wrote a little NUnit test fixture class to run tests against the web page. The key object is the DefaultSelenium class that is created in the SetUp() method:

/// <param name="serverHost">the host name on which the 
/// Selenium Server resides</param>
/// <param name="serverPort">the port on which the 
/// Selenium Server is listening</param>
/// <param name="browserString">the command string used 
/// to launch the browser, e.g. "*firefox", "*iexplore"
/// or "c:\\program files\\internet explorer\\iexplore.exe"</param>
/// <param name="browserURL">the starting URL including 
/// just a domain name.  We'll start the browser pointing at
/// the Selenium resources on this URL,
/// e.g. "http://www.google.com" would send the browser to 
/// "http://www.google.com/selenium-server/SeleneseRunner.html"</param>
public DefaultSelenium(String serverHost, int serverPort, 
String browserString, String browserURL)
{
  this.commandProcessor = new HttpCommandProcessor(serverHost, 
serverPort, browserString, browserURL);
}
using System;
using NUnit.Framework;
using Selenium;
 
namespace SeleniumTarget
{
    [TestFixture]
    public class WebPageTester
    {
        DefaultSelenium selenium;
 
        [SetUp]
        public void SetUp()
        {
            // 4444 is the default port for the Selenium Server
            selenium = new DefaultSelenium("localhost", 4444, "*iexplore", "http://localhost");
            selenium.Start();
        }
 
        [TearDown]
        public void TearDown()
        {
            // Make sure the Selenium environment is cleaned up after each test
            selenium.Stop();
        }
 
 
        [Test]
        public void CheckTheTitle()
        {
            selenium.Open("http://localhost/SeleniumTarget/TestPage1.htm");
 
            Assert.AreEqual("Selenium Target Page",
                            selenium.GetTitle(),
                            "Check the title of the browser");
        }
 
 
        [Test]
        public void ClickButton1ChangesText1FromHelloToGoodbye()
        {
            selenium.Open("http://localhost/SeleniumTarget/TestPage1.htm");
            Assert.AreEqual("Hello",
                            selenium.GetValue("text1"),
                            "Initial Value");
 
            selenium.Click("button1");
 
            Assert.AreEqual("Goodbye",
                            selenium.GetValue("text1"),
                            "Value after clicking button1");
        }
 
        /// <summary>
        /// Check that the options of a <select></select> element are
        /// as expected.  Finds the <select> element by using an xpath expression
        /// </summary>
        [Test]
        public void Select1Values()
        {
            selenium.Open("http://localhost/SeleniumTarget/TestPage1.htm");
            string locator = "xpath=//select[@testid='select1']";
 
            string[] options = selenium.GetSelectOptions(locator);
 
            Assert.AreEqual(new string[] {"North", "West", "South", "East"},
                            options,
                            "Values in the select1 dropdown");
        }
    }
}

It’s a trivial example, but it’s a start. 

What I don’t know –

  • What’s the best way to handle the Selenium Server process?  How do you guarantee it’s up when you start your tests?  I’m thinking some kind of Windows service wrapper
  • How do you integrate Selenium with CruiseControl.Net to get it into your Continuous Integration strategy?  I can’t find much on the web about this yet.  You can run Selenium from NUnit for developer testing, but that isn’t a very desirable answer from a tester’s perspective.  For a couple of reasons, our thinking is to wrap the Selenium manipulation inside a FitNesse DoFixture.  We already know how to integrate FitNesse tests within a CC.Net build and it’s convenient to run all the tests together.  We’re also hoping that the DoFixture tests will be easier to understand and that we can hide some of the web page details behind the fixture.

Other Tools

We’re looking primarily at Selenium, but we’re also considering WATIR (Ruby based tool) and Sahi (I don’t know much about it, but it looks strong).  One of our colleagues is experimenting with a Ruby based DSL for testing another web product using WATIR as the core that looks promising.  I’ve also been playing with the Ruby driver for Selenium with an eye towards creating a testing DSL for our application.   

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Test Driven Development. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://bryancook.net Bryan

    Jeremy,

    You might want to check out the Selenium Toolkit for .NET. It provides:

    - bundled installer for Selenium IDE / RC
    - NUnit addin that starts/stops selenium-server for test runs
    - simplified configuration and extensibility.

    Love to hear your feedback, have a great day!

  • http://www.contentdisha.com Content Writer

    It does seems to hang occasionally. My concern would be cleaning up dangling instances of the Selenium exececutable

  • http://www.crestechsoftware.com Sandya

    I have three Questions
    Q1:–When I run my test using Test Runner in Selenium
    then the links on the site “www.crestechsoftware.com” are not clicked automatically by the selenium tool

    Can any body tell me why it happens..

    Q2 : When I tried to see the run log of my test suite, it displayed “blank page” giving nothing..

    Can anybody tell me why this happens..

    Q3: Can we save our test execution results in Selenium..

  • sandhya

    Hi

    I m new to selenium …this is first day into it.
    Kindly help me step by step guide to it… or share some path where i can find the same. As i m new to this forum as well n wont be accessing it throughout, so pls drop in the reply as soon as possible.
    Thanks

  • Monal

    Hi ,

    This is Monal, am currently working on Selenium RC and I came across a problem.
    The Issue is “How do we make selenium recognize pop ups?” Do we have any spl addins?
    Plz let me know some details on the same.

    Thanks in Advance

    Monal

  • ak

    The last piece of code was the one I have been looking for a long time. – Thank You.

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    @Khaled,

    I think all you need to do is specify “tHead” in your xpath expression. There shouldn’t be anything special to do.

  • khaled

    hello jeremy.
    i have a probleme with selenium.
    how to make To recover a String in the thead of a table and not tbody with xpath

    thank you for your help(exuse my english…i know is very bad)

  • http://www.incisif.net Frederic Torres

    Jeremy,

    As you mentioned “Several tools have used COM to drive Internet Explorer (IE) with varying degrees of success” and the ‘varying degrees of success’ may be part of the problem.

    Selenium RC provides a solution to implement cross browsers testing.
    But does not give you a direct access to the DOM (because of the Java proxy).

    I am wondering if you can automate a drag and drop with Selenium or handling the Save As dialog.

    If you are not a Microsoft hater and are .NET friendly, you could have a look at InCisif.net.

    See how I implemented a drag and drop with InCisif.net at my post:

    http://blog.incisif.net/2007/04/02/watin-incisifnet-and-drag-and-drop.aspx

    To finish where you started:

    The Agile development community is still struggling on how to automated testing solutions for web development.
    I really like to see a kind FIT/FITNESSE for Web User Interfaces.

    Let us try to find some good solutions without starting another war of religion.

  • Alex

    You can also check SWExplorerAutomation (SWEA) from http://webiussoft.com. SWEA records, replays and generates test script code in C# or VB.NET. SWEA also supports automation of complex Web applications developed with AJAX.

  • jmiller

    Thanks for the link Allan, I’ll definitely give it a long look.

  • Allan With

    Hi Jeremy

    Thanks for showing how painlessly simple and powerful Selenium .NET testing can be! I have been looking for something like this for a long time.

    I saw this description on how to integrate Selenium with CruiseControl on OpenQA’s website:
    http://wiki.openqa.org/display/SEL/Integrating+Selenium+And+CruiseControl.Net

    I haven’t actually tried it, but it might just do what you are looking for.

  • http://dhtmlkitchen.com/ Garrett

    In Java 5, pass a collection and compare against a selenium.getSelectOptions(locator);

    /**
    * Checks equivalence of a collection and a String[].
    * The String[] is from selenium.getSelectOptions().
    * Usage: Test a dataset against an option list.
    *
    * @param titles
    * @param options
    */

    public void assertOptionsPresent(ArrayList titles, String[] options) {

    assertTrue(
    titles.size() == options.length &&
    titles.containsAll(Arrays.asList(options))
    );
    }

  • Sachin

    Jeremy,

    I don’t quite see how you will have dangling instances of selenium server, since it won’t let you open up another instance at the same port.

    Of course we still need a test for whether the server has hanged.

    I was working on another idea; didn’t quite finish it yesterday. You could have a NUnit base class which sends a web request to
    http://host:port/selenium-server/SeleneseRunner.html

    If it comes back with a web response, then we can be sure there is a listener on that port and it is servicing requests. If not, it will come up with a web exception or will time out, in which case the test should fail with an appropriate error.

    Would have been good if there was a way to do this declaratively in CC.NET.

  • Sachin

    Just tried opening up another instance at the same port. Jetty handles this quite gracefully and shuts down, which is nice.

  • Sachin Rao

    I don’t see how you would have floating instances, because it won’t be able to open up the web server at the same port (assuming you’re trying to restart it at the same port).

    But we definitely need a test to see if the server has hanged or not.

    I was trying something else out, but couldn’t quite finish it yesterday. You could have a base NUnit class, which on setup would open a web request to
    http://host:port/selenium-server/SeleneseRunner.html.

    If it times out, or if it comes back with an WebException, then the server isn’t available or it is not servicing requests anymore. In which case the test should fail with an appropriate exception. If we do get a web response, it means all is well and we can proceed with our tests.

    Its not the best of solutions.

  • jmiller

    Sachin,

    It does seems to hang occasionally. My concern would be cleaning up dangling instances of the Selenium exececutable

  • Sachin

    You could perhaps use the NAnt task to run this. Dunno how you would explicitly stop this process, though. Then again, why would you want to?

  • http://www.thejoyofcode.com/ Josh

    Crikey Jeremy, you’re not having a lot of luck keeping these posts viewable! Do you work on a billion pixel monitor?

    :D

    It’s a shame because I’m just not going to read this post now.

    Josh

  • Jason

    I have used HtmlUnit in the Java world where NUnitASP is used in .NET. HtmlUnit has excellent JavaScript support and allowed us to perfrom intergation level testing in our web application. It was not perfect but is definitely worth a look.

  • David Kemp

    Also, and perhaps more relevantly, be interested to hear how you get on with Sahi.
    I’m currently working with some (‘classic’) ASP apps, and any tools I can find to help with any automated testing at all would be wonderful.

  • David Kemp

    This post is displaying funny:
    http://flickr.com/photos/b3ardman/146770254/
    Firefox 1.5.0.3 on Windows XP SP2

  • Richard

    Very interesting!

    We use Selenium a lot but only in “Test Runner mode”. This is mostly because we haven’t found a good way to combine the IDE (http://www.openqa.org/selenium-ide/) and it’s recording capabilities with the Driven mode. To start with the IDE lacks a custom C# format for c# (http://wiki.openqa.org/display/SIDE/Contributed+Custom+Formats).

    Selenium Driven mode also struggles with https – right?

    //Richard