Jeffrey Palermo [MVP]

In reality, developers don’t like to do much testing.  Developers aren’t testers.  We typically will write code while making certain assumptions about variables and, objectively knowing the expected behavior of the syntax, we might run it once and call it done.  Typically, bugs hide where the code wasn’t rigorously tests or in paths that weren’t tested at all.  I think everyone agrees that without testing, the software will have bugs (and often even after testing).  I heard of a crazy management quote that is very sad: “If you have to test it, you aren’t building it right.”  I sure am glad that manager wasn’t involved with automobile development!  In reality, we need testing

Types of common developer testing

• Actually running the code written
• Simple console application or winforms test harnesses
• Running the application through the UI locally or on a development server.

Types of testing

• Unit testing
• Integration testing
• Acceptance testing
• Load testing
• Performance testing
• Security testing
• Exploratory testing
• And many more

Approaches

• Do all manual testing with or without the help of small tools.
• With this approach, the cost is high because for every run of the test, human time is required.  For every release, every test case must be repeated.
• This approach doesn’t scale.  When more testing is needed, that directly calls for more human time.  Human time is very expensive compared to computing time.
• Automate all testing.
• The cost is low for about 80%, but then goes up.
• Some types of tests are hard or impossible to automate effectively, such as security, exploratory, and concurrency testing.  This type of testing needs more human attention and isn’t easily repeatable.
• To fully succeed at this approach, testers need to be highly skilled at scripting.
• A pragmatic approach – automate the testing that is easiest.
• Unit testing
• Integration testing
• Acceptance testing

Unit testing

What is a unit test?  From Wikipedia:  A “. . . unit test is a procedure used to validate that a particular module of source code is working properly”.  A unit test is a very unique type of test with some fundamental constraints.  If a test violates a constraint, it ceases existence as a unit test.

Characteristics of a unit test

• A unit test isolates and tests a single responsibility.

• A unit test isolates a failure.
• A unit test simulates other collaborators using various methods such as fake objects.
• A unit test can run and pass in any environment.

When an automated test reaches out to a file, registry key, database, web service, external configuration, environment variable, etc, it becomes an integration test and is polluted with dependency on its environment.  At that point, the binary can no longer be sent to another location with the expectation that test continue to pass. 

Unit testing is hard.  It requires constant vigilance to ensure application code is loosely-coupled.  Because a unit test must test a piece of code in isolation, the code must be designed so that collaborators can be easily separated.  In short, calling out to a static method from inside a constructor is a tight-coupling.  This type of code will hinder unit testing.

Integration testing

What is integration testing?  An integration test is a developer test that combines several units of code and tests the interaction among them.  A good integration strategy will include tests along all integration boundaries.  If a unit of code is tested at the unit level and then tested during interaction with all collaborators, the developer has confidence that the code will work well when the entire system is assembled.  Integration testing is the second step in a “test small, test medium, test large” strategy.  Larger integration tests are certainly useful, but large tests alone don't help to pinpoint a problem if they fail. 

Characteristics of a good integration test

• It only includes a few classes.  It typically focuses on a single class and then includes collaborators.  If a class has 3 collaborators, the test would include 4 classes (the first class and its 3 collaborators).  This test would aim to fake the collaborators twice removed so that the focus on the test can remain narrow.  This is a subjective rule, and the developer should use good judgment regarding how many classes to include in an integration test.
• It is completely isolated in setup and teardown.  Should not be environmentally sensitive.  If the test requires a file to be on the drive, then setup for the test should place the required file there first.  If the test needs data in a database, the setup should insert the required records first.  It should not depend on any environment setting being present before the test is run.  It should run on any developer workstation.
• Must run fast.  If it’s slow, the build time will suffer, and you will run fewer builds.
• It must be order-independent.  If an integration test causes a subsequent test to fail, it is indicative of a test not owning its own environmental setup.

Acceptance testing

What is acceptance testing?  An evaluation of acceptance criteria on the system to ensure the system meets the customers’ needs.  Before a piece of software can be written the software team must have acceptance criteria.  This is often called requirements.  Acceptance testing seeks to execute the requirements on the system for a pass/fail result.

Characteristics of a good acceptance test

• Easily understood by non-developers as well as the customer.
• Easily created by non-developers as well as the customer.
• When it passes, the customer is assured that the system behaves as needed.
• Very expressive using jargon of the business domain.
• Is repeatable.

Automated acceptance testing is the final big gain in the suite of automated tests.  The acceptance tests actually define requirements in an objective and executable way.  If a tester finds a bug, he can write an acceptance test that fails in the presence of the bug.  When the bug is fixed, the test will pass.  Acceptance tests become an important part of a regression test suite over time.

Stay tuned for more detailed information on the following testing topics:

• Unit testing
• Integration testing
• Acceptance testing