Recently,
we needed to isolate our integration tests from our unit tests. They were mixed
in the same test project and we wanted to avoid the cost of executing
integration tests while executing our unit tests. Integration tests takes
longer to be executed because they are identified as tests that are accessing external
systems, such as DB, files, registry, web service…
At first,
it looked like a daunting task because we needed to know which tests were triggering
file access and we have thousands of tests. More precisely, we needed to know
which test method from our test assembly were depending on the class System.IO.File or depending on the
method System.Xml.XmlDocument.Load(String).
Hopefully, we realized that a simple CQL query
would list all integration tests.
SELECT METHODS FROM ASSEMBLIES "NDepend.Test.Unit" WHERE
(IsUsing "System.IO.File" OR IsUsing "System.Xml.XmlDocument.Load(String)")
This CQL
query works well because the condition IsUsing
XXX matches direct and indirect use. It means that if Test1() is calling a method that is calling a method that … that is
calling XmlDocument.Load(String),
then Test1() will be identified as
an integration test. Notice that if an interface is used to mock a service
(such as file or DB access), the IsUsing
condition won’t see what’s hidden behind the interface. Also, if our
application was using DB, we would have needed some more DB conditions such as
… WHERE … IsUsing "System.Data.Common.DBCommand" OR IsUsing "System.Data.Common.IDBConnection" OR …
We also
used the Dependencies Matrix to see which test was using which file method. A
blue cell tagged with X means that the test method (in abscissa) is using the file
method (in ordinate) with a minimum path depth of X. For example, we can see
that the test Test_CheckNDependDeployment_Normal()
is using Files.Exist() with a depth
of 4 (see the generated graph below the matrix):
Because NDepend relies on static analysis (i.e it
is analyzing the code structure, not the run-time call graph), we got a few
false positives. Indeed, some test methods were statically depending on a file
access method without accessing it at run-time. However, we decided to consider such
tests as integration tests, this way we can keep our simple criteria to sort
between unit and integration tests.