Create a Testing DSL with FitNesse and Selenium (Part 1)

My team uses FitNesse extensively to create automated acceptance tests for our systems.  Recently we’ve added Selenium RC running from within FitNesse to our toolbox.  There’s a lot of buzz lately over the creation of Domain Specific Languages (DSL), driven in no small part to create a syntax for the application code that is readable by the business partners and testers.  We accomplish the same goal (or at least try to) by using FitNesse fixtures to effectively create a DSL for our applications that our tester can use to specify and verify the expected behavior of the system.  FitNesse is both the “language” parser and a reporting framework that can already be plugged into NAnt and CC.Net, so there’s no real need to create custom interpreters.  Take that you Ruby cool kids with your “missing_method” metaprogramming magic!


Most of my last week and a half has been spent building the “Mother of all Automated Tests.”  The system under test was basically a new messaging workflow between our company and a partner company for something that we call a “Bundle.”  All told, the entire Bundle pipeline includes:



  • A web application on our side to start the  Bundle workflow and approve or reject work from our partner
  • An extension to our messaging broker component that runs as a polling windows service
  • A web service, that we’re building, installed on server at our partner’s that will receive notifications of a new Bundle for them to process
  • A web service on our side that will receive a pair of notifications from the partner, including the finished product.  This web service enforces a wide range of business rules

The communication pipeline is fairly complex, there isn’t a user interface for much of it, the web service messages are non-trivial, and the validation rules are extensive.  Creating a FitNesse grammar for us and our tester to simulate the communication has been a lifesaver.  Using our DSL we can quickly throw a lot of different scenarios at the system in a repeatable manner.


The DoFixture is the Lynchpin


FIT (and FitNesse) has always had the ability to express tests as declarative tables.  It works great when you can stay within the lines of x number of inputs leads to y outputs, but it breaks down for more complex scenarios like, well, basically anything in an enterprise application.  It’s also a mess to maintain state between fixture tables because all you have are static members.  All of that changed because now we have the DoFixture.  I think I’ve said before in this blog that I didn’t particularly care much for FIT style testing until we got the DoFixture from Rick Mugridge’s FitLibrary extension. 


The DoFixture allows us to create custom textual languages to express tests.  In other words, jagged tables.  I’ll leave it for a later post, but the DoFixture allows us to nest other fixture tables to do declarative assertions.  Think of the DoFixture as an interpreter that enables us to quickly create grammars without getting bogged down by writing our own parsing and reporting functionality.  Here’s an example straight out of our tests on the Bundle workflow.



    public class CreateBundleScreenFixture : DoFixture


    {


        public void CreateBundleFromToAsAWithInvoiceTotalWithAttachment(


            string senderId,


            string receiverId,


            string communicationType,


            decimal invoiceTotal,


            string fileName)


        {


            // call into your application’s service layer with the inputs above


        }


    }


That’s the Fixture code, it’s nothing special, just a method. Let’s move onto the representation in a FitNesse table. In FitNesse’s particular version of Wiki markup, “!” is a directive to start a new table. New table cells are started just by delimiting the contents with a “|” character. Just to avoid some unpleasantness, always remember to close each line with a pipe bar and no trailing spaces.  I’m going to make all of the FitNesse sample in Wiki text because that’s what you’ll be editing (and the blog engine will screw up the table formatting).

	!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |


The DoFixture parses this command by treating the table row as an array of cells.  Starting with the first cell, DoFixture iterates through every other cell, concatenates the text together, and removes all of the white space.  The DoFixture uses this concatenated string as the method name, which is located through reflection (lots and lots of reflection).  Back to the other cells that got skipped over, DoFixture assumes that these cells are the arguments to the method in order.  The DoFixture is able to determine the type of the argument from the method signature and coerce the types.  In this case the DoFixture automatically converts the string “1018.23″ into a decimal type before pushing it into the invoiceTotal argument of the CreateBundleFromToAsAWithInvoiceTotalWithAttachment() method.  FitNesse itself in .Net handles all of the normal primitive types, including DateTime values.  In an annoying oversight, the .Net version misses enumeration types.  If anybody is interested, I can show some code to plug into FitNesse to modify the way FitNesse parses and handles any particular type.


So that’s how you create an action in a DoFixture.  FitNesse will of course fail the test if the action blows up with an exception, but you also need to make assertions about the outcome of the action.  The simplest thing to do is create a method that returns a Boolean.  The DoFixture will automatically treat any method with a Boolean return value as an assertion.  Without any arguments, that might look like:



        public bool TheBundleWasCreatedSuccessfully()


        {


            return _lastBundleSucceeded;


        }

	!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To |
Receiver1 | as a | Fax | With Invoice Total |
1018.23 | With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |

Fancier assertions can be made by using the Check keyword.  Let’s add some grammars to test the Status of the newly created Bundle object.



        public void AliasTheLastBundleAs(string bundleName)


        {


            // store the Id of the last Bundle created in the FitNesse fixture


            // as bundleName


        }


 


        public string TheBundleStatusOfIs(string bundleName)


        {


            // look up the Status field from the Bundle aliased


            // as [bundleName] and return


        }

	!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |
| Alias the Last Bundle As | TestBundle1 |
| Check | The Bundle Status of | TestBundle1 | Is | New |

By starting the test row with “Check,” I’ve directed DoFixture to assert that the value of the TheBundleStatusOfIs(“TestBundle1″) method is the value in the last cell.  When the test is executed, FitNesse will color the last cell green if it succeeds, or red with an NUnit-like “expected blah, actual blah” text description.  One of the best advantages to FitNesse testing is the visceral red/green output.  It’s usually pretty easy to spot the test failures (diagnosing the cause, is alas, as difficult as always). 


More complex assertions usually call for a nested Fixture table.  I’ll demonstrate a couple examples in later sections.  If you can’t already guess, I love the DoFixture.


Environment Settings


At some point FitNesse and Selenium have to know some environment settings.  You can accomplish this through the normal configuration techniques, but I’ve started to put a lot of this type of information in the FitNesse tests themselves.  FitNesse allows for a “SetUp” test page that will be included inside each test page in a test suite.  I used a little DoFixture that looks something like this:



    public class MySystemEnvironmentFixture : DoFixture


    {


        // Yes, I do realize that I’m using a public static field


        // It’s okay — in this context anyway


        public static string ApplicationRoot = “http://localhost/MyApp/”;


        public static string WebServer = “http://localhost”;


        public static string BrowserName = “*firefox”;


 


        private static DefaultSelenium _selenium;


 


 


        public void TheWebServerIs(string webServer)


        {


            WebServer = webServer;


        }


 


        public void TheApplicationIsRootedAt(string root)


        {


            ApplicationRoot = root;


        }


 


        public void TheBrowserIs(string browserName)


        {


            BrowserName = browserName;


        }


    }


Inside the SetUp page I have a table like this:

	!|MySystemEnvironmentFixture|
| The Web Server Is | http://MyTestServer/ |
| The Application Is Rooted At | http://MyTestServer/TheTestingRoot/ |
| The Browser Is | *iexplore |

In one place, I can quickly shift the FitNesse fixtures from pointing from a local installation on my workstation, to the development servers, or finally to the testing servers.  If you’ll notice the third line, I can also switch the browser engine that Selenium is using (IE, Firefox, Opera, etc.).  That’s awfully important in the age of AJAX/Web 2.0 tricks with multiple browsers.


Integrating Selenium into a DoFixture


Finally, let’s pull Selenium into the mix to drive the web pages.  While you can use Selenium with its own FIT-like driver, or though NUnit, I like running Selenium from within FitNesse for acceptance tests.  I like this approach because we get a unified reporting and execution infrastructure for both UI and Service Layer tests.  Plus, and this cannot be underestimated, I can use regular FitNesse fixtures to setup and verify application state before and after the Selenium execution in a single self-contained test.


We’re using the .Net driver for Selenium RC.  I wrote a brief introduction to using Selenium RC from .Net that explains the basic mechanisms.


The first thing to do is create a new Selenium session and make it accessible to all of the Fixture’s in a test.  I’m working on an assumption that we will normally write a separate FitNesse fixture for each non-trivial screen, and share the Selenium driver object in a static member somewhere in memory.  I’m going to start by adding some infrastructure to the MySystemEnvironmentFixture just to provide access to a DefaultSelenium object.



        // Create a new Selenium session


        private static void resetSelenium()


        {


            // Use the ApplicationRoot and BrowserName fields


            _selenium = new DefaultSelenium(“localhost”, 4444, BrowserName, ApplicationRoot);


            _selenium.Start();


        }


 


        // Provide access to the currently executing Selenium session


        // to other screen fixtures


        public static DefaultSelenium SeleniumDriver


        {


            get


            {


                if (_selenium == null)


                {


                    resetSelenium();


                }


 


                return _selenium;


            }


        }


 


        // Start a Selenium session, probably call this


        // in SetUp


        public void StartApplication()


        {


            resetSelenium();


        }


 


        // Shut down the Selenium session and free the


        // resources.  Either do this in TearDown or at


        // the end of every, single test.


        public void CloseApplication()


        {


            if (_selenium != null)


            {


                _selenium.Stop();


            }


        }


Since every call to “Start Application” will spawn a new browser instance, you should definitely remember to call “Close Application” at the end of each test or in the TearDown. Unless you don’t mind out of memory exceptions on your build server;)

	!| MySystemEnvironmentFixture |
| Start Application |
| Close Application |

Now, let’s create a little fixture to for a screen. In our Bundle workflow user interface we have a screen that displays all of the data about a single Bundle and has buttons to perform common actions on a Bundle like “Reject” or “Approve.”



    public class BundleDetailsFixture


    {


        // Elements on the screen


        private const string STATUS_LABEL = “xpath=//*[@testid='status']“;


        private const string APPROVE_BUTTON = “xpath=//*[@testid='status']“;


        private const string REJECT_BUTTON = “xpath=//*[@testid='status']“;


 


        private const string URL = “BundleDetails.aspx”;


 


        DefaultSelenium _selenium;


 


        public BundleDetailsFixture()


        {


            // Go grab the current Selenium session object


            _selenium = MySystemEnvironmentFixture.SeleniumDriver;


        }


 


        public void OpenTheBrowserToBundle(string bundleName)


        {


            // Find the Id for the Bundle aliased as [bundleName]


            long bundleId = 0;


 


            string url = string.Format(“{0}{1}?Id={2}”, MySystemEnvironmentFixture.ApplicationRoot, URL, bundleId);


            _selenium.Open(url);


        }


 


        public string TheBundleStatusIs()


        {


            return _selenium.GetText(STATUS_LABEL);


        }


 


        public bool TheApproveButtonIsEnabled()


        {


            return _selenium.IsEditable(APPROVE_BUTTON);


        }


 


        public bool TheRejectButtonIsEnabled()


        {


            return _selenium.IsEditable(REJECT_BUTTON);


        }


 


 


        // Click the Reject button, and then click Cancel


        // on the confirmation popup box


        public void ClickTheRejectButtonAndCancel()


        {


            _selenium.ChooseCancelOnNextConfirmation();


            _selenium.Click(REJECT_BUTTON);


        }


 


        public void ClickTheApproveButton()


        {


            _selenium.Click(APPROVE_BUTTON);


        }


 


 


    }


The simplest case is to call the service layer first to create a Bundle. Next we’ll look at the Bundle Details screen to confirm the status of the Bundle, then approve the Bundle and check the status again.


!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |
| Alias the Last Bundle As | TestBundle1 |

Now, call into the screen. I’m assuming that the Selenium session was created in the FitNesse SetUp
By the way, you can quite happily mix prose with your testing tables

!| BundleDetailsFixture |
| Open the Browser to Bundle | TestBundle1 |
| Check | The Bundle Status Is | New |
| The Approve Button Is Enabled |
| Click the Approve Button |
| Check | The Bundle Status Is | Approved |


That’s pretty simple stuff.  Just about everything you need to check or manipulate on a web page is accessible from the DefaultSelenium object.  If it’s not, you can send over your own Javascript to be evaluated in the browser.  I haven’t bumped into anything that required that technique yet.  Selenium itself will largely take care of page transitions and postbacks for you.  If need be, you can direct Selenium to wait for a page transition with a timeout, or poll for a certain condition if you’re doing heavy AJAX work.


One of the other advantages for wrapping Selenium within FitNesse is to abstract away the page structure from the test writer.  The test writer, who may or may not have knowledge to the application’s internals, can focus on the English language definition of the test and defer the xpath or id location of elements to the developers.  Because ASP.Net likes to munge up element id’s of nested controls, we’ve adopted a convention of using an arbitrary “testid” attribute on an element to be the Selenium key for testing.  We can find an element with a testid anywhere on the page by using a locator like “xpath=//*[@testid='the test id'].”


The FitNesse wrapper gets really handy when you want to perform multiple steps in a single sentence, like repeatedly filling in a form in the same steps but with different input. Here’s a cool example I haven’t needed yet, but might soon. With most testing frameworks you have to setup an expectation for what to do when the next popup box shows up before you define the step that creates spawns the popup button. That’s potentially confusing, and it certainly makes your testing harder to understand.  With the DoFixture we can write a better grammar for that case:

	!| BundleDetailsFixture |
| Open the Browser to Bundle | TestBundle1 |
| Check | The Bundle Status Is | New |
| Click the Reject Button and Cancel |
| Check | The Bundle Status Is | New |

Wrapping Up


I wrote this in a hurry and probably missed quite a few details.  If you have any questions, or better yet, suggestions, feel free to leave a comment and I’ll get to it quickly for once.


It’s just not that hard.  The hardest part of the giant end-to-end test was really the deployment scripts.  All I’ve shown here is the mechanics of simple cases.  In the next installment, I’ll talk about the pattern I use to load data in FitNesse tests, and using nested Fixture’s within a DoFixture to do more complex data entry and assertions.  I’ll also try to get a post in on lessons learned and best practices, but I’m going to wait until the big end-to-end testing infrastructure is completely verified;)

About Jeremy Miller

Jeremy is the Chief Software Architect at Dovetail Software, the coolest ISV in Austin. Jeremy began his IT career writing "Shadow IT" applications to automate his engineering documentation, then wandered into software development because it looked like more fun. Jeremy is the author of the open source StructureMap tool for Dependency Injection with .Net, StoryTeller for supercharged acceptance testing in .Net, and one of the principal developers behind FubuMVC. Jeremy's thoughts on all things software can be found at The Shade Tree Developer at http://codebetter.com/jeremymiller.
This entry was posted in Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://www.ilogos-ua.com Maxim

    Hello!
    Do you consider cooperation with dedicated Php developer or dedicated development team based on beneficial terms?
    Let me introduce myself, my name is Maxim Slobodyanyuk.
    I am a founder and CEO of http://www.ilogos-ua.com (iLogos – outsourcing software development team, based on Ukraine)

    We have pretty good PHP developer, his name is Alex. He has 6 years of web-development experience includes: project architecture creation, design of user interfaces, multilayer web-applications. 3 year experience in object-oriented development: creation of database management system, graphics processing applications, web-site processing.
    If you are interested in dedicated PHP developer – feel free to contact me (skype: getstar777, icq: 217580424, gtalk: slobod777).

    Technologies and tools

    PHP, HTML, XML, CSS, JavaScript, AJAX, SQL, Apache, Asm, VB, FoxPro, Clarion, Tcl/Tk, 1C., SVN, Mantis, BAZAAR, CVS

    Education
    Master of computer science
    1999-2004 East-Ukrainian National University named after Volodymyr Dahl
    Faculty of management. Economical cybernetics. (specialty “Information Systems in management”)

    About him:
    • Has well team lead skills
    • Abstract, innovative and creative thinking
    • Highly organized, attention to detail and maintain a professional demeanor at all times
    • Ability to work independently and as part of team
    • Good communication and writing skills (fluent English)

    Portfolio
    We can represent you fulfilled projects by individual inquiry.

    What I would like to offer you:
    • Communications via skype\icq\gtalk\etc
    • 40 hours \ week (full-time)
    • daily \ weekly reporting (if required)
    • if required I can sign the contract for this cooperation + NDA
    • The responsible approach – is guaranteed

  • http://www.davidpeterson.co.uk David Peterson

    Noooo, don’t write scripts in FitNesse. It will be a maintenance headache. See: http://www.concordion.org/Technique.html for an alternative way of writing the tests.

  • Jayyusi

    Hello,
    How can I know in my test class, that it is currently the last line in the fitnesse table aktiv (by testing…)?
    or the count of the rows and the currently activ row?
    thanks
    Jayyusi

  • Jayyusi

    Hello,
    How can I know in my test class, that it is currently the last line in the fitnesse table aktiv (by testing…)?
    or the count of the rows and the currently activ row?
    thanks
    Jayyusi

  • Charles Chidi

    Hi Jeremy, I am trying to introduce fitnesse and Selenium IDE to my current web test project. Issue is I have only used Fitnesse and Selenium once, very sparingly and not quite there yet!! Now I have to install the various software and find a way to add my tests in. I have already written manual tests for these and tried testing using visual studio, but that would not allow for validation on the pages..as it was built with flash technology. Where do istart? the instructions around the various products are a bit too technical and I am looking for a simple approach. Wondering what Fixtures to use, how to link the browsers to Fit and where to enter the data to be validated. Can i get a step by step guide? f this would help,…pretend I am stupid and totally ignorant to technology!! Kindly point me to the right direction!!! Many thanks

  • Sakthivel

    Hi Jeremy,
    I am new Fitnesse with Selenium.I want to execute my selenium scripts with fitnesse table values by conditional checking.
    Can you help me in this.

    Thanks in Advance,
    Sakthi.K

  • Jim

    Hi, Ive been working with Selenium-RC and Fitnesse.

    I’ve reach a point where I need to go to other domains, for this I can only use -proxyInjectoMode, bc the *chrome and *custom doesnt work, I get java errors in the Terminal, the Test doesnt work.

    But when I use -proxyInjectionMode something goes wrong with the website encoding, bc the site is in Spanish and chars like ñ, á, é, …. doesnt show, but crap.

    Do you have any idea why it is?

    Thanks

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    You can do it behind the scenes in another method. You can also use symbols to store it, but I’ve never tried it with the DoFixture and doubt it works.

  • Priya

    i want to return a value from a function and store it in a variable, is it possible to do this in DoFixture?
    my fitnesse table looks like this

    this is fetched from database
    |Get Country |IE|Threshold|${threshold}|

    |Enter Payment Amount |${threshold}|

    I want to fetch a value from database and store it in a variable in Fitnesse.

    Please let me know if there is any solution

    Priya

  • http://ww.luxoft.com/fit Michael vax

    If you are looking on an alternative solution to Fitnesse that keeps tests in a simple file system, have a look at FITpro. FITpro provides plugins for Eclipse and Visual Studio that allow creation and execution of FIT tests inside IDE. Tests are stored as HTML files which simplifies integration with source control and automated build. Refactoring is also made easier by having your application code, fixtures, and fit tests under the same IDE.

  • http://kasajian.com Kenneth Kasajian

    Proposing something called “PhraseFixture” — a better DoFixture() for use with FIT and FitNesse Acceptance Testing Framework.

    The powerful DoFixture() in FitLibrary simulates English like specification. The PhraseFixture proposal takes this to the next level.

    Please see article here:
    http://codeproject.com/KB/library/PhraseFixture.aspx

    I’d like to get some feedback.

  • Nidhi

    Hi Jeremy,

    In our company we are also trying to use the same approach, but standardise the selenium related fixtures so that, they can be used by any test.

    Please can you tell me how do we run the same set of fit+sel commands for different values everytime.
    Implying is is possible to loop through fitnesse lines n number of times

  • http://codebetter.com/blogs/jeremy.miller Jeremy D. Miller

    Rubecca:

    I don’t know how to do that. That’s been a long standing gripe about FitNesse.

  • Rubecca

    Hi Jeremy, i’m jus getting started with integrating selenium and fitness.
    My fitnesse pages get saved by default in th FitNesseRoot directory.
    Please can you tell me how to save my testpages outside Fitneese and yet have them run.

  • http://www.autoriginate.com Patrick Lightbody

    Jeremy,
    I’m glad to see that Selenium RC is working out so well for you. My company, Autoriginate, is actually the original creator of Selenium RC. It came from work we were doing for our commercial products, HostedQA and AutoQ.

    The one thing that is a general bummer about about FIT is that refactoring of steps and creating your own DSL involves modifying the fixture classes. This is something Rick Mugridge and I have been emailing back and forth about, seeing if there was a way you could get all the benefits of the DSL without requiring someone crack open the fixtures every time.

    I started Autoriginate with the goal of addressing that very issue. They way we do it is by simply allowing for very basic refactoring support (Extract Macro, Introduce Parameter, Delete Macro, etc) via a simple UI. In short, when viewing a test, click on a few steps and they will be highlighted blue. Click the extract Macro button and viola, you’ve got a new DSL command. You can then analyze your other tests for opportunities to replace parts of them with that new command, so you can quickly refactor and create very nice high level commands, like “ClickTheRejectButtonAndCancel” or “CreateNewAccount”.

    If you’d like to see more, you can sign up for a free trial (don’t worry – no sales folks will bother you, you’ll just get access right away) and try extracting a few commands.

    Patrick Lightbody
    Founder and CEO
    Autoriginate, Inc.
    Creators of HostedQA and AutoQ
    Creators of Selenium Remote Control
    Founders of OpenQA

  • http://www.thejoyofcode.com/ Josh

    Thanks Jeremy. It still doesn’t fit in my aggregator (sharpreader) but at least I can just open this page and read it now.

    Josh

  • jmiller

    Try it now if you would. I wrapped the FitNesse tables that are in a

     tag
  • Brian

    The scrolling does make it very hard to read your blog. Can you wrap some of the longer code?

  • http://www.thejoyofcode.com/ Josh

    Sorry Jeremy – but the scrolling is doing my nut in!

    http://www.thejoyofcode.com/Not_everybody_has_superhuge_monitor.aspx