TestNG - A Flexible Java Test Framework

TestNG - A Flexible Java Test Framework

By Rad Widmer, OCI Senior Software Engineer

July 2010

Introduction

TestNG is a Java testing framework that is particularly well-suited for use on large projects. It provides especially strong support for writing functional and integration tests as well as unit tests. TestNG key features include:

TestNG uses Java annotations extensively. This means no reliance on class inheritance or method naming conventions as in JUnit 3. While JUnit 4 also uses annotations and shares some features with TestNG, there are significant differences in philosophy and implementation. JUnit strives to keep tests isolated from one another, while TestNG is more flexible, allowing various types of dependencies between tests. The key difference is the context in which the tests run.

TestNGJUnit
All test methods in the same class execute in the same instance of the class     Each test method executes in its own class instance      
Easy to share resources among tests Sharing resources among tests limited and not encouraged    

The Basics

TestNG can be invoked in a number of ways - from the command line, from within an IDE such as Eclipse, IntelliJ IDEA, or NetBeans, from the testng Ant task, or from Maven. In all cases, a suite XML file (also referred to as a testng.xml file) is used to define the test suites. Although it is not required, a suite XML file provides the most power and flexibility in configuring and selecting which tests to run. Here's a simple command line example, which runs the suite defined by testng.xml (assuming the testng jar file and any classes used during testing are on the classpath):

    java org.testng.TestNG testng.xml

There are several useful optional arguments, including

The Suite XML (testng.xml) file

A testng.xml file defines a single test suite. A suite contains one or more test elements, each of which specifies a set of test classes. The tests which make up the test element can be specified in one or more of the following ways:

Here is an example suite which defines a test named CoreTests that includes classes ClassATest and ClassBTest, plus all test classes in the package com.ociweb.jnb.testng.core and any subpackages (because the package name ends with ".*"). In addition, the group long-tests is excluded from the test.

  1.    <suite name="Suite" parallel="false">
  2.       <test name="CoreTests">
  3.         <classes>
  4.           <class name="com.ociweb.jnb.testng.ClassATest"></class>
  5.           <class name="com.ociweb.jnb.testng.ClassBTest"></class>
  6.         </classes>
  7.         <packages>
  8.           <package name="com.ociweb.jnb.testng.core.*"></package>
  9.         </packages>
  10.         <groups>
  11.           <run>
  12.             <exclude name="long-tests"></exclude>
  13.           </run>
  14.         </groups>
  15.       </test>
  16.     </suite>

For more details on the suite XML file, an HTML version of the DTD schema is available at http://testng.org/dtd/.

Annotations

Any class can be turned into a TestNG test class by adding at least one TestNG annotation. The @Test annotation is used to designate the test methods in a class. It can be applied to individual methods, as in the following example:

  1.   public class ExampleMethodAnnotations {
  2.         @Test
  3.         public void checkSomething() {}
  4.  
  5.         @Test
  6.         public void checkSomethingElse() {}
  7.  
  8.         public void notATestMethod() {}
  9.     }

The @Test annotation can also be applied to a class, in which case all public methods of that class will be test methods, unless they are annotated by one of the @BeforeXXX or @AfterXXXconfiguration annotations described below.

  1.    @Test
  2.     public class ExampleClassAnnotation {
  3.         public void checkSomething() {}
  4.  
  5.         public void checkSomethingElse() {}
  6.     }

Configuration Annotations

Configuration annotations can be applied to methods to perform some action either before or after specific events:

For example, the methods in the class TestBeforeAfter are executed in the following order:

    beforeSuite
    beforeTest
    beforeClass
    beforeMethod
    test1
    afterMethod
    beforeMethod
    test2
    afterMethod
    afterClass
    afterTest
    afterSuite
  1.     public class TestBeforeAfter {
  2.         @BeforeSuite
  3.         public void beforeSuite() {}
  4.  
  5.         @AfterSuite
  6.         public void afterSuite() {}
  7.  
  8.         @BeforeTest
  9.         public void beforeTest() {}
  10.  
  11.         @AfterTest
  12.         public void afterTest() {}
  13.  
  14.         @BeforeClass
  15.         public void beforeClass() {}
  16.  
  17.         @AfterClass
  18.         public void afterClass() {}
  19.  
  20.         @BeforeMethod
  21.         public void beforeMethod() {}
  22.  
  23.         @AfterMethod
  24.         public void afterMethod() {}
  25.  
  26.         @Test
  27.         public void test1() {}
  28.  
  29.         @Test
  30.         public void test2() {}
  31.     }

Groups

It's natural to want to place tests into groups. For example, you may have a core group of tests which should be run before any files are checked in to the source code repository. You could also have groups for different types of tests, such as integration tests and performance tests. Tests which cover different features could also be in different groups. A "broken tests" group can be useful for tests which are known to be broken, and won't be fixed for a while.

In TestNG, groups are used to control execution of tests, both by including or excluding tests from a suite, and by controlling the order of tests and whether tests are skipped. A test method can be a member of any number of groups. Test methods can be assigned to groups at the class level (all test methods in the class are members of the class-level groups), on a per-method basis, or a combination of both.

An example of assigning groups at the method level:

  1.   public class MethodLevelGrouping {
  2.         @Test(groups={"a"})
  3.         public void a() {}
  4.  
  5.         @Test(groups={"b"})
  6.         public void b() {}
  7.  
  8.         @Test(groups={"a", "b"})
  9.         public void ab() {}
  10.     }

To assign groups at the class level, add a @Test annotation to the class. This turns all public methods in the class into test methods (unless they have a @BeforeXXX or @AfterXXX annotation). These methods also acquire any annotation elements (such as groups) defined by the class-level @Test annotation. In the following example, testB1 and testB2 are test methods belonging to group b, and method testBC belongs to groups b and c.

  1.    @Test(groups={"b"})
  2.     public class ClassLevelGrouping {
  3.         public void testB1() {}
  4.  
  5.         public void testB2() {}
  6.  
  7.         @Test(groups={"c"})
  8.         public void testBC() {}
  9.     }

Test Dependencies

Tests can depend on other methods or groups. This effects test execution order and can cause tests to be skipped depending on the results of other tests. The execution order rules are as follows:

There are two types of dependencies, hard (the default) and soft. This determines whether tests are skipped or not.

In this example, test depTest depends on methods pre1 and pre2. depTest always runs after pre1 and pre2, and will be skipped if either pre1 or pre2 fail. Note that pre1 and pre2 can be executed in any order.

  1.  public class DependsOnMethods {
  2.         @Test(dependsOnMethods={"pre1", "pre2"})
  3.         public void depTest() {}
  4.  
  5.         @Test
  6.         public void pre1() {}
  7.  
  8.         @Test
  9.         public void pre2() {}
  10.     }

Here's an example where a number of tests depend on an external server and some environment configuration. We want to avoid executing the server-dependent tests if the server is down. One way to do this is do create test methods which verify that the server is working and the environment is configured correctly. For convenience, these methods are placed in an init group, and the server-dependent tests depend on this group. Then, if the server is down, the server-dependent tests will be skipped. This has the benefit of potentially saving a lot of time (no waiting for the server to time-out for each test), plus the error report is more accurate, allowing one to focus more quickly on the actual failures. In this example, the init group methods are placed in their own class. Alternatively, all the methods could be placed into one class, though this would require annotations on each method.

  1.  @Test(groups={"init"})
  2.     public class InitTests {
  3.         public void checkEnvironment() {}
  4.  
  5.         public void checkServer() {}
  6.     }
  7.  
  8.     @Test(dependsOnGroups={"init"})
  9.     public class ServerDependentTests {
  10.         public void sdtest1() {}
  11.  
  12.         public void sdtest2() {}
  13.     }

Adding Flexibility with Method Parameters

Test and configuration methods can have parameters. Parameter values can be assigned in two ways - by using the @Parameters annotation, or by using DataProviders.

@Parameters

Here's an example of using the @Parameters annotation to assign a server name and port number.

  1.     @Parameters({"server-name", "port"})
  2.     @BeforeTest
  3.     public void setupServer(String serverName, int port) {
  4.         this.serverName = serverName;
  5.         this.port = port;
  6.     }

The parameter values are assigned in the suite XML file (note that the name attributes must match the names listed in the @Parameters annotation):

  1.   <test name="ParametersTest">
  2.     <parameter name="server-name" value="test-server "/>
  3.     <parameter name="port" value="1234"/>
  4.     ...
  5.   </test>

Parameter elements can be placed under the  or  elements. Suite parameters apply to all tests unless they are overridden by a parameter of the same name under a element.

While the @Parameters approach is easy to use, it has some limitations. Parameter types are limited to simple types such as String, int, and double. Also, the parameter values are only assigned once, so it is not possible to invoke the same method with multiple sets of parameter values. DataProviders overcome these limitations.

DataProviders

A DataProvider is a method annotated with @DataProvider which returns either an Object[][] or Iterator. In either case, the inner Object[] contains the parameters for a single invocation of a test method. The number and types of elements in each row must match the number and types of parameters in the test method. Here's a simple example:

  1.    @DataProvider(name = "indexOfProvider")
  2.     public Object[][] indexOfTestGenerator() {
  3.         return new Object[][] {
  4.                 { -1, "something", "x" },
  5.                 { 4, "something", "thing" }, };
  6.     }
  7.  
  8.     @Test(dataProvider="indexOfProvider")
  9.     public void testStringIndexOf(int expect, String s1, String s2) {
  10.         assertEquals(expect, s1.indexOf(s2));
  11.     }

In this example, the test method will be invoked twice, first with parameters {-1, "something", "x"}, then with parameters {4, "something", "thing"}.

DataProviders can be particularly useful when implementing data-driven tests - tests where it is necessary to perform the same checks with a variety of input data sets. Typical ways of handling these types of tests include:

DataProviders can provide a more elegant solution by helping keep your test methods small, and by separating the test data generation from the actual tests.

Parallel Execution

TestNG supports running tests in multiple threads. Use this feature to speed up execution of thread-safe tests. By default, all tests run in a single thread. To use multiple threads, use the parallel and thread-count attributes of the  and  tags. The parallel attribute can be set to the following values:

The thread-count attribute is used to set the number of threads in the thread pool. If parallel or thread-count attributes are specified on a  tag, their values take precedence over the values.

Here's one way to define a suite where parallel is "methods" by default, but is overridden by a test which contains methods which should not be run in parallel:

  <suite name="Suite" parallel="methods" thread-count="10">
  ...
  <test name= "not-parallel" parallel="false">

Reports

By default, TestNG produces an HTML report and a TestNG specific XML file. In addition, XML files are produced which are compatible with the Ant JUnitReport task. This is handy for integrating into existing build frameworks, but lacks any TestNG-specific data. It is also possible to produce custom reports by implementing an IReporter reporter class. This class is called when all the test suites have completed, and it is passed the test results for all tests in the suites.

Summary

TestNG provides good support for managing large test suites, and provides solutions to common problems which arise particularly when writing functional and integration tests. This makes it a natural fit for many projects, where it can make it easier to write tests, more efficient to run the tests, and easier to interpret the results.

References

secret