Reality
In reality, developers don’t like to do much testing. Developers aren’t testers. We typically will write code while making
certain assumptions about variables and, objectively knowing the expected behavior
of the syntax, we might run it once and call it done. Typically, bugs hide where the code wasn’t
rigorously tests or in paths that weren’t tested at all. I think everyone agrees that without testing,
the software will have bugs (and often even after testing). I heard of a crazy management quote that is
very sad: “If you have to test it, you aren’t building it right.” I sure am glad that manager wasn’t involved
with automobile development! In reality,
we need testing
Types of common
developer testing
- Actually
running the code written - Simple
console application or winforms test harnesses - Running
the application through the UI locally or on a development server.
Types of testing
- Unit
testing - Integration
testing - Acceptance
testing - Load
testing - Performance
testing - Security
testing - Exploratory
testing - And
many more
Approaches
- Do all
manual testing with or without the help of small tools. - With
this approach, the cost is high because for every run of the test, human
time is required. For every
release, every test case must be repeated. - This
approach doesn’t scale. When more
testing is needed, that directly calls for more human time. Human time is very expensive compared
to computing time. - Automate
all testing. - The
cost is low for about 80%, but then goes up. - Some
types of tests are hard or impossible to automate effectively, such as
security, exploratory, and concurrency testing. This type of testing needs more human
attention and isn’t easily repeatable. - To
fully succeed at this approach, testers need to be highly skilled at
scripting. - A
pragmatic approach – automate the testing that is easiest. - Unit
testing - Integration
testing - Acceptance
testing
Unit testing
What is a unit test?
From Wikipedia: A “. . . unit
test is a procedure used to validate that a particular module of source
code is working properly”. A unit test
is a very unique type of test with some fundamental constraints. If a test violates a constraint, it ceases
existence as a unit test.
Characteristics of a
unit test
- A unit
test isolates and tests a single responsibility.
- A unit
test isolates a failure. - A unit
test simulates other collaborators using various methods such as fake
objects. - A unit
test can run and pass in any environment.
When an automated test reaches out to a file, registry key,
database, web service, external configuration, environment variable, etc, it
becomes an integration test and is polluted with dependency on its
environment. At that point, the binary
can no longer be sent to another location with the expectation that test
continue to pass.
Unit testing is hard.
It requires constant vigilance to ensure application code is
loosely-coupled. Because a unit test
must test a piece of code in isolation, the code must be designed so that
collaborators can be easily separated.
In short, calling out to a static method from inside a constructor is a
tight-coupling. This type of code will
hinder unit testing.
Integration testing
What is integration testing?
An integration test is a developer test that combines several units of
code and tests the interaction among them.
A good integration strategy will include tests along all integration
boundaries. If a unit of code is tested
at the unit level and then tested during interaction with all collaborators,
the developer has confidence that the code will work well when the entire
system is assembled. Integration
testing
is the second step in a “test small, test medium, test large”
strategy. Larger integration tests are certainly useful, but
large tests alone don’t help to pinpoint a problem if they fail.
Characteristics of a
good integration test
- It
only includes a few classes. It
typically focuses on a single class and then includes collaborators. If a class has 3 collaborators, the test
would include 4 classes (the first class and its 3 collaborators). This test would aim to fake the
collaborators twice removed so that the focus on the test can remain
narrow. This is a subjective rule,
and the developer should use good judgment regarding how many classes to
include in an integration test. - It is
completely isolated in setup and teardown.
Should not be environmentally sensitive. If the test requires a file to be on the
drive, then setup for the test should place the required file there
first. If the test needs data in a
database, the setup should insert the required records first. It should not depend on any environment
setting being present before the test is run. It should run on any developer
workstation. - Must
run fast. If it’s slow, the build
time will suffer, and you will run fewer builds. - It
must be order-independent. If an
integration test causes a subsequent test to fail, it is indicative of a
test not owning its own environmental setup.
Acceptance testing
What is acceptance testing?
An evaluation of acceptance criteria on the system to ensure the system
meets the customers’ needs. Before a
piece of software can be written the software team must have acceptance
criteria. This is often called
requirements. Acceptance testing seeks
to execute the requirements on the system for a pass/fail result.
Characteristics of a
good acceptance test
- Easily
understood by non-developers as well as the customer. - Easily
created by non-developers as well as the customer. - When
it passes, the customer is assured that the system behaves as needed. - Very
expressive using jargon of the business domain. - Is
repeatable.
Automated acceptance testing is the final big gain in the
suite of automated tests. The acceptance
tests actually define requirements in an objective and executable way. If a tester finds a bug, he can write an
acceptance test that fails in the presence of the bug. When the bug is fixed, the test will
pass. Acceptance tests become an
important part of a regression test suite over time.
Stay tuned for more
detailed information on the following testing topics:
- Unit
testing - Integration
testing - Acceptance
testing
I believed in NUnitASP, until I had to access a datagrid. Datagrids alone are no problem, but try accessing the contents of a datagrid row nested inside a DIV tag with the pages viewstate turned on….
I believed in NUnitASP, until I had to access a datagrid. Datagrids alone are no problem, but try accessing the contents of a datagrid row nested inside a DIV tag with the pages viewstate turned on….
I would actually take a look at WebAii here: http://www.artoftest.com, You can use .net to automate rich web scenarios using IE or FireFox. You can also automate you javascript unit testing.
Also check out TestRegions, a concept to help reduce test automation maintenance.
I disagree NUnitAsp is too fragile unless you use pure XHTML compliant pages. Sometimes the pages are slow, but the tool is robust. I have never considered this standard (I had to look up what XHTML is) when writing NUnitAsp tests. I am not knocking Selenium since I think it is a good tool, but NUnitAsp is pretty darn good, too.
MVP pattern is the same as ModelViewController I’m guessing?
I am just starting a new development project, and was looking at that. I’ve got the same problem with the Pattern bloggers as I had with the TDD bloggers
Too much theory, not enough samples. However, it seems that for once, opening my fat mouth has been a good thing…
NUnitAsp seems to be what I’m looking for, at least for a start. I’ll check out selenium in the next couple of days and will be looking at the links you sent.
As I said, I know I need to refactor and get my code out of my presentation, I was just totally unclear on how that could possibly help me test. Simply by making a new call from an event to a new method doesn’t tell me anything about how that helps me to test. I know I have to do it, but without samples of testing code that works in my environment, it doesn’t get me to step 2.
Hopefully the links you included not only tell me how to refactor a standard vanilla db driven aspx page into an MVC pattern, but also how to test that my header, detail and footer lines of my datagrid are generating correctly.
Carlton, I owe you a beer if we ever meet up just for mentioning NUnitAsp. At the very least, I can use it to tell if all my pages are loading after doing something stupid to a class.
I’m still trying to figure out how I missed NUnitAsp and Selenium in my google searches. Whatever did I do to you, oh great and powerful Google?
Bob,
The very first thing to do is to look *hard* at the MVP pattern, and you really need to get over the intepridation over mock objects or stubs. The actual WebForms machinery is difficult, but not impossible, to test. Get as much code as possible away from your screen. The datagrid event handlers should simply delegate to a controller that can be tested easily.
I’d recommend Selenium over NUnitASP for testing. NUnitASP is too fragile unless you can really enforce pure XHTML compliant pages.
Model View Presenter links:
http://www.objectmentor.com/resources/articles/TheHumbleDialogBox.pdf#search='the%20humble%20dialog%20box by Michael Feathers
http://www.martinfowler.com/eaaDev/ModelViewPresenter.html
http://mikemason.ca/2006/02/15/
http://codebetter.com/blogs/jeremy.miller/archive/2006/02/01/137457.aspx
http://codebetter.com/blogs/jeremy.miller/archive/2006/02/16/138382.aspx
http://codebetter.com/blogs/jeremy.miller/articles/129546.aspx
For unit testing close to the UI, I’m going to give an example of the Model-View-Presenter pattern. Reading up on it now would be beneficial. This allows testing all the way up to the GUI layer (which is made to be very small). Because GUI needs eyeballs to ensure it looks good it will always need manual testing, but we can still unit test GUI behavior.
I’ll take a look at NUnitAsp. And I’ll be shooting you an email also!
Thanks!
Bob, if you are writing Dot.Net code, there is NUnitAsp out there for writing exactly the types of tests you are talking about. I have TONS of example code I would be more than willing to share with anyone – nettletonc_at_saic_dot_com.
Bob,
Great point. I was frustrated with UI and database testing as well when I started automated testing. I’m going to make a few more posts on automated testing, and I’ll cover these. I’m not going to even touch on TDD. I’m only going to talk about automated testing. Stay tuned.
Just got set to receive email updates to this post.
Bob
Jeff,
I’ve seen everyone and their mother, and sometimes younger siblings, put together a blog post about TDD, or Testing, or “What is testing” or some other title. Most of them are pretty good, at least the ones that I can read all the way through.
Every once in a while, I even see one that actually appears to have a bit of practical value, so I go and get those tools and try to apply them to my Web Application.
At which point, with everything installed and all the little “Hello World” samples running great, I now ask: “OK, now what? How do I test my real world app?” Deafening silence. I never seem to be able to find a true, fully fleshed out Web App sample, with real DB access, and DATE’s in the database.
I understand generally that I need to refactor, such that my methods can be called independently. However, most of the time when TDD or any kind of automated testing would help me, the problems are showing up in a db call or in screen layout. Simple, or even complex, methods and functions that I write are almost NEVER the problem. Sure I could put together tests for those methods, but that is mostly a waste of my time.
I’ve been trying to get into TDD for over a year now, but without having seen a great article or a great tool that can help me test my web app, it’s all just light reading.
If you and the other ( Mostly great and well-informed. PLEASE do not think this is a rant against you or TDD. ) folks who write about TDD want the world to embrace it, take us through a small web dev process, show us what will get tested and how. Be thorough. Don’t ever use the “I’ll leave this as an excercise for the user.” garbage sentence.
Again, I am most interested in screen layout/results ( like datagrids having and displaying the correct data ) and how to test Event functions. I have no clue at all how to test my Datagrid1_ItemDataBound or dg2_ItemCommand from a harness. I would LOVE to. ( And NO, my datagrids are not called Datagrid1… )
OK, this bit can be taken as a friendly rant…
Enough with the theory all you TDD gurus. Show me real world examples, including some thorough guidance on how to set up a “TDD version” of my database. Real examples of how to return it to it’s starting state. When NOT to return it to start state so I can figure out where the newly introduced bug is.
Especially show me how in heck to test server-side-control events, and show me how to do it without using the word “stub” or in fact using a stub.
Any test that doesn’t do what a user might do is not thorough enough for me to trust. I can’t be the only one who feels frustrated by the lack of examples. Heck, I’ll even be happy if anyone can send me a link that contains some good hard examples of the stuff I’ve mentioned.
I can’t be the only one who wants all of this stuff and doesn’t have the time to figure out all the intricate details on my own.
I just noticed you’re a c# person. I know both c# and vb.net, so I don’t even care which language, but you’d probably reach the largest audience by doing this sample in both languages.
I hope some of the other TDD gurus read this, and I hope someone has the time to do it and publicize the fact that they did it.
TDD is taking over most of the blogs I get in my feeds. As a web dev, it’s getting a bit old for me.
Ayende,
integration testing is a very broad term, and end-to-end tests are certainly integration tests. I believe the value of an integration test starts by testing just a few classes together and then testing more on top of that. I believe it’s valuable to test in the small before testing from end-to-end.
Your defination for Integration testing is different than what I know.
What do you call to a test that validate system behavior end to end? From the UI (or just below) to the database and back?
This is usually what I would call integration tests, and that doesn’t run on every build (although I would probably run it on every CI build.