JUnit: Not Just Another Pretty Assert

JUnit: Not Just Another Pretty Assert

by Charles Sharp, Principal Software Engineer

March 2014

What's in your JUnit tests?

JUnit 4 was released Feb 16, 2006. With its release came annotations and the ability to write JUnit tests in a more flexible fashion than the setUp, test, and tearDown of 3.8. JUnit 4 took three and a half years to show up after the 3.8.1 release but since then there have been ten more releases -- 4.11 came out in December, 2012 and 4.12 is in the works.

So here we are ten releases and eight years later. And I wonder, how much has changed in the way you use JUnit? Are you using the Hamcrest matchers with assertThat or perhaps assumeThat? Rules? Categories? And how about those parameter-driven tests using Theories, eh?

This article reviews some of JUnit 4's capabilities to refresh your memory and perhaps remind you of some of the advantages to using something, well, besides assert. (You are using asserts, aren't you?)

We start with a quick review of the Assert class and the Hamcrest matchers, then move to the Assume class. After that, there are sections for Categories, Parameterized testing using Parameter and Theory, and we end up with Rules.

The sections are intended to be stand-alone, so feel free to check out the ones that interest you most, but take at look at all of them. They're all good tools.

Getting started

This article assumes that you’re familiar with creating and running JUnit 4 unit tests. That is, you know how to create tests and test suites and how to use the @Before and @After fixtures for them. Of course, you need to know how to run tests from your IDE or the command line.

If those are not correct assumptions, then I would recommend the Getting Started page on the JUnit wiki as a great place to, well, get started.

Or, if you’re still using JUnit 3.8, you might want to take a look at an earlier OCI SETT article on migrating to JUnit 4, Migrating from JUnit 3 to JUnit 4. It's definitely time to be using JUnit 4.

Assert

The Assert class is the workhorse of JUnit and is the class JUnit testers are most familiar with. Most JUnit assert signatures are similar in nature. They consist of an optional message, an expected instance or variable and the actual instance or variable to be compared. Or, in the case of a boolean test like True, False, or Null, there is simply the actual instance to be tested.

The signature with a message simply has an initial parameter with a message string that will be displayed in the event the assert fails:

assert<something>(“Failure Message String”, <condition to be tested>);

The form without a message has the same arguments but omits the reason message:

assert<something>(<condition to be tested>);

For example, here are two asserts that compare the values of two long variables to each other:

public static void assertEquals(String message, long expected, long actual);
public static void assertEquals(long expected, long actual);

A listing of the available asserts and their details are found in the JUnit Assert Javadoc.

The JUnit site also has a quick guide that shows some of the available asserts in action at JUnit Wiki Assertions page.

An interesting assert that differs from the others is assertThat. It features both a different approach and signature:

assertThat([java.lang.string Reason], T actual, org.hamcrest.matcher <T> matcher)
assertThat(T actual, Matcher<? super T> matcher)

Rather than an expected instance or value, the actual instance of T to be tested comes first and is compared to the assertions contained in the Hamcrest matchers.

Hamcrest matchers, though bundled with JUnit, are a separate project. Hamcrest is a general purpose matching library for Java, Python, Ruby, Objective-C, PHP, and Erlang. Use http://hamcrest.org as the place to start to learn more about the overall Hamcrest project. A great place to start for an understanding of what Hamcrest matchers offer the JUnit tester is The Hamcrest Tutorial. This short tutorial provided by the Hamcrest folk uses JUnit 3.8 but gives a good overview of the categories of matchers available and also shows how to write a custom matcher.

Once you have a good feel for what the Hamcrest matchers provide, Marc Philipp has created an excellent quick reference of the 1.3 matchers which can be found at Hamcrest 1.3 Quick Reference

In JUnit, the Hamcrest matchers provide composability of intent and readability, especially in the error messages. For simple cases, this is not very apparent. Take for example, an assert() failure when comparing two longs:

  1. long L1 = 93;
  2. long L2 = 92;
  3. assertEquals("Off by one again!", L1, L2);

This failure produces:

java.lang.AssertionError: Off by one again! expected:<93> but was:<92>

which, as failure indications go, is fine. The stack trace following the error message accurately pinpoints the location of the error.

Here is the same failure using assertThat("Off by one again!", L2, equalTo(L1));

java.lang.AssertionError: Off by one again!
Expected: <93L>
but: was <92L>

It is a bit nicer seeing the data type but it is essentially the same error message as the other assert and required more typing on the tester's part. However. Once the assertions become more complicated, assertThat shows its strength:

  1. String expected = "Another";
  2. String actual = "Started well, did One thing, Ended well.";
  3. assertThat("So it goes", actual,
  4. allOf(startsWith("Started"), endsWith("well."), containsString(expected)));

When this fails, you receive the message:

java.lang.AssertionError: So it goes
Expected: (a string starting with "Started" and a string ending with "well." and a string containing "Another")
   but: a string containing "Another" was "Started well, did One thing, Ended well."
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:865)
at com.ociweb.AlwaysPassesTest.test(AlwaysPassesTest.java:40)
... <rest of stack omitted>

This is a great description of what part of the Matcher failed and where.

In this case, using asserts to do this is more work. You can either use three asserts, such as:

  1. assertTrue("So it starts", actual.startsWith("Started"));
  2. assertTrue("So it ends", actual.endsWith("well."));
  3. assertTrue("So it goes", actual.contains("Another"));

When the third assert fails, you receive a message to the effect:

java.lang.AssertionError: So it goes
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at com.ociweb.AlwaysPassesTest.test(AlwaysPassesTest.java:35)
...

This fairly common approach allows you to see which assertion fails but makes for longer tests and more typing. Or, you can combine all the asserts to sort of look like the assertThat version with matchers:

  1. assertTrue("So it goes",
  2.   actual.startsWith("Started") &&
  3.     actual.endsWith("well.") &&
  4.       actual.contains("Another"));

This produces an error message like:

java.lang.AssertionError: So it goes
  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at com.ociweb.AlwaysPassesTest.test(AlwaysPassesTest.java:37)
  ...

This now seems to be a bad idea since the tester has no idea which condition of the assert actually failed.

While we're composing with matchers, the failing matcher test above can be made to pass in a couple of ways. One would be the anyOf matcher:

  1. String possible = "One thing";
  2. String alsoPossible = "or Another";
  3. String actual = "Started well, One thing, Ended well.";
  4. assertThat(
  5.   "So it goes", actual,
  6.   allOf(startsWith("Started"), endsWith("well."),
  7.     anyOf(containsString(possible), containsString(alsoPossible))));

The either matcher also works:

  1. assertThat(
  2.   "So it goes", actual,
  3.   allOf(startsWith("Started"), endsWith("well."),
  4.     either(containsString(possible)).or(containsString(alsoPossible))));

To summarize, when a test is working with a single result and wants to state a single expectation of that result, then plain asserts work fine. But when the test needs to express multiple expectations of that result, matchers are a clear and concise way to do that while maintaining the ability to quickly determine the point of failure should a test fail.

Assumptions

You’ve probably heard that it’s best not to work on assumptions so here is a testing tool JUnit gives you to ensure your tests don’t.

Both Asserts and Assumes stop when a test fails and move on to the next test. The difference is that a failed Assert registers the failure as a failed test while an Assume just moves to the next test. This permits a tester to ensure that conditions, some of which may be external and out of control of the tester, are present as required before a test is run.

There are four varieties of Assumes: one to check a boolean condition, one to check that an exception has not occurred, one to check for null objects, and one that can take a Hamcrest matcher. As seen in the Assert section above, the ability to take a Hamcrest matcher is a gateway to testing flexibility.

The assumeFalse(boolean b) and assumeTrue(boolean b) both work like a boolean assertFalse(boolean b) or assertTrue(boolean b). The assumes check to see if their boolean argument is false or true. If the boolean matches expectations, the test continues. Otherwise the test is halted.

assumeNotNull(Object... objects) does a simple check of the Object arguments to ensure they are not null. If any are null, the test is halted.

assumeNoException(Throwable t) is used to ensure that an exception was not thrown during an operation. This is useful when an operation such as opening a file or connecting to a server could throw an exception. Said another way, you can use this assumption to ensure required resources are successfully obtained prior to testing.

assumeThat(T actual, Matcher matcher) is the version of assume that uses a Hamcrest matcher. This method is similar in form and function with assertThat(T actual, Matcher matcher) and permits a test to concisely check multiple conditions before proceeding with a test. In addition, it is useful for selecting the tests to run when using Theories in parameter-driven testing as shown in the Parameterized Testing section below.

Categories

Categorizing things is a very natural activity and unit tests are no exception. There are regression tests, fast tests, or tests that only pertain to Platform X or Release V. The Category annotation permits a tester to classify their unit tests. The example given in the Categories Javadoc gives an example of fast or slow tests, something any tester can relate to.

Regardless of the categories you use to classify your tests, Categories are used to either specify which group of tests to run or which group of tests to not run. Inclusion is more specific than the case of exclusion. To show this, we'll use the Javadoc example of SlowTests and FastTests. Let’s say you have some methods annotated as follows:

In one class containing tests, there are some SlowTests and some tests that may not be fast, but are not slow:

  1. ...
  2. public class BunchOfMostlySlowUnitTests {
  3.  
  4. @Category(SlowTests.class)
  5. @Test
  6. void firstSlowTest() { ... }
  7.  
  8. @Category(SlowTests.class)
  9. @Test
  10. void secondSlowTest() { ... }
  11.  
  12. @Test
  13. void firstNotSlowTest() { ... }
  14. ...

In another test class, we know that all the tests are in the FastTests category so we just decorate the class:

  1. ...
  2. @Category(FastTests.class)
  3. public class BunchOfFastUnitTests {
  4.  
  5. @Test
  6. void firstFastTest() { ... }
  7.  
  8. @Test
  9. void secondFastTest() { ... }
  10. }
  11. ...

Finally, there is a test class with tests that are neither FastTests nor SlowTests:

  1. ...
  2. public class BunchOfUnitTests {
  3. @Test
  4. void firstNeitherTest() { ... }
  5. @Test
  6. void secondNeitherTest() { ... }
  7. ...

The Category runner extends the Suite runner so the setup to run Category tests looks just like a Suite:

  1. @RunWith(@Categories.class)
  2. // @IncludeCategory(SlowTests.class)
  3. // @ExcludeCategory(FastTests.class)
  4. @SuiteClasses( { BunchOfMostlySlowUnitTests.class, BunchOfFastUnitTests.class, BunchOfUnitTests.class } )
  5. public class CategoryTestSuite { }

Finally we’re ready to talk about Including and Excluding tests.

If there are no @IncludeCategory or ExcludeCategory decorations on the @SuiteClasses, all the tests in the suite classes are run -- just like a @Suite. No surprises there.

If you uncomment the @IncludeCategory shown in line 2, only the tests specifically decorated with@Category(SlowTests.class) will run, that is to say, the firstSlowUnitTest() and secondSlowUnitTest() will be executed.

Contrast that to what happens if instead you uncomment the @ExcludeCategory shown in line 3. In this case, all tests but those specifically decorated with @Category(FastTests.class) will run. That means firstSlowUnitTest(),secondSlowUnitTest(), and the two NeitherTests methods will run. All of that to say the use of @ExcludeCategory is more inclusive in practice.

JUnit does not come with any Categories pre-defined. A tester builds them as needed. So how is that done? The element used on the @IncludeCategory, @ExcludeCategory, and @Category annotations is a Java class. The class can be either a class or an interface. An easy and usual approach is to create an empty interface with the name of the desired category. Need a Regression Test category? Use: public interface RegressionTest {}. You're now ready to use@Category(RegressionTest.class). Very easy.

One more thing about the @Category annotation. Its element is actually an array of Java classes. That means you can use the @Category to assign multiple categories to a single test, as in: @Category({ SlowUnitTest.class, RegressionTest.class }) { ... }.

Categories can be very useful. In the realm of Continuous Integration, the need to have the unit tests perform and move on is imperative. On the other hand, sometimes a tester needs to ensure that the Pause method really did. The use of Categories can make these conflicting needs less painful.

Parameterized Testing

JUnit provides two different runners to drive parameterized testing, Parameters and Theory. Which to use? If the tester can easily and completely provide the data needed to test, then testing with Parameters is a good way to go. If, on the other hand, the test automatically generates data or obtains it from a non-deterministic source, Theory works better.

Parameters

To run a Parameterized tests, you need three things: a static method to generate the test data, the tests (of course), and either a constructor to receive and store the data or a set of decorated variables to receive the data.

Here's an example of using the Constructor to store the data:

  1. package com.ociweb;
  2. import static org.hamcrest.CoreMatchers.*;
  3. import static org.junit.Assert.*;
  4.  
  5. import java.util.Arrays;
  6. import java.util.Collection;
  7.  
  8. import org.junit.Test;
  9. import org.junit.runner.RunWith;
  10. import org.junit.runners.Parameterized;
  11. import org.junit.runners.Parameterized.Parameters;
  12.  
  13. @RunWith(Parameterized.class)
  14. public class ParameterizedExample {
  15.   private String testData;
  16.   private String expectedResult;
  17.  
  18.   @Parameters
  19.   public static Collection<Object[]>  generateData() {
  20.     return Arrays.asList(new Object[][] {
  21.       { "JAVA", "JAVA" },
  22.       { "CLOJURE", "CLOJURE" },
  23.       { "C++", "FUN" } }
  24.     );
  25.   }
  26.  
  27.   // Constructor is invoked once for every element in the Collection
  28.   // annotated with @Parameters
  29.   public ParameterizedExample(String testData, String expectedResult) {
  30.     this.testData = testData;
  31.     this.expectedResult = expectedResult;
  32.   }
  33.  
  34.   @Test
  35.   public void testTheData()
  36.   {
  37.     ClassUnderTest cut = new ClassUnderTest();
  38.     String actualResult = cut.doImportantThingsToBeTested(this.testData);
  39.     assertThat(actualResult, is(expectedResult));
  40.   }
  41. }

When the test is run, the generateData method(line 19) will supply the data. Note that the reason this happens is the @Parameters annotation on the method. The data supplied by this method will then be iterated over and each iteration will be fed to the constructor (line 29). The constructor in turn uses the data to set the fields to be tested(lines 15 and 16). After the construction is complete, the test or tests are then run with that data. This continues until the end of the data collection is reached.

Here is the same example done without a constructor. In this example, the data values are injected directly into the fields to be used in the test.

  1. ...
  2. @RunWith(Parameterized.class)
  3. public class ParameterizedExample {
  4.   @Parameters
  5.   public static Collection<Object[]>  generateData() {
  6.     return Arrays.asList(new Object[][] {
  7.       { "JAVA", "JAVA" },
  8.       { "CLOJURE", "CLOJURE" },
  9.       { "C++", "FUN" } }
  10.     );
  11.   }
  12.  
  13.   @Parameter(value=0)
  14.   public String testData;
  15.   @Parameter(value=1)
  16.   public String expectedResult;
  17.  
  18.   @Test
  19.   public void testTheData()
  20.   {
  21.     ClassUnderTest cut = new ClassUnderTest();
  22.     String actualResult = cut.doImportantThingsToBeTested(this.testData);
  23.     assertThat(actualResult, is(expectedResult));
  24.   }
  25. }

Here the constructor has been removed. The data generator method is the same. But the fields that receive the data have been decorated with the @Parameter annotation. The value element determines the index of the data to be applied to the variable.
Note Well! The data fields are now public not private like they were in the previous example that used a constructor to set the values.

One final thing for Parameters. There is a nice optional element for the @Parameters annotation that allows you to have a nice run title for each run. For example, if the example above is run in Eclipse Kepler, the output will show something like:

Parameters No Name

But a name element can be added to the Parameters annotation like this:

  1. @Parameters(name = "DataSet #{index}: testData:{0} expected:{1}")
  2.     public static Collection<Object[]>  generateData() {

When the test is run now, the output looks like:

Parameter Name

In the name element, {index} refers to the current iterator index in the Collection. The {<N>} entries refer to the parameter values. If no name element is given the default is: (name = {index}) to match previous JUnit releases.

More information and another example can be found at the JUnit wiki Parameterized tests page.

Theories

On the other side of the parameter-driven testing house is Theory. To run parameter-driven tests with Theories, you need two things: data points and theory tests. The data points can be supplied using either a static method that generates test data in an array and is marked with the @DataPoints annotation or there can be fields annotated with@DataPoint or there can be a mixture of both as in the following example. You also need the tests (of course) decorated with the @Theory annotation. The Theories.class runner will exercise all Theory tests that take the same argument type provided by a DataPoint.

In the following example, the DataPoint Strings are sent one-by-one to the test methods taking a String argument. Since a Theory test really doesn't know beforehand what the data will be like, there is an assumeThat done to ensure that this test is to be run on the data. Likewise for the methods taking an int or a long.

  1. package com.ociweb;
  2.  
  3. import static org.hamcrest.Matchers.*;
  4. import static org.junit.Assume.assumeThat;
  5.  
  6. import org.junit.experimental.theories.DataPoint;
  7. import org.junit.experimental.theories.DataPoints;
  8. import org.junit.experimental.theories.Theories;
  9. import org.junit.experimental.theories.Theory;
  10. import org.junit.runner.RunWith;
  11.  
  12. @RunWith(Theories.class)
  13. public class TheoriesExample {
  14.     @DataPoints
  15.     public static String[] data() {
  16.         return new String[] { "JAVA", "SCALA", "CLAN" };
  17.     }
  18.  
  19.     @DataPoint
  20.     public static int intToNotTest = 0;
  21.  
  22.     @DataPoint
  23.     public static int intToTest = 1;
  24.  
  25.     @DataPoint
  26.     public static long longToNotTest = 19;
  27.  
  28.     @DataPoint
  29.     public static long longToTest = 43; // off by 1, again!
  30.  
  31.     @Theory
  32.     public void hasAnA(String dataToTest) {
  33.         assumeThat(dataToTest, containsString("A"));
  34.         // Perform tests for strings with an "A"
  35.     }
  36.  
  37.     @Theory
  38.     public void needsToContainW(String dataToTest) {
  39.         assumeThat(dataToTest, containsString("W"));
  40.         // Perform tests for strings with a "W"
  41.     }
  42.  
  43.     @Theory
  44.     public void is5CharactersLong(String dataToTest) {
  45.         Integer testStringLength = new Integer(5);
  46.         assumeThat(dataToTest.length(), equalTo(testStringLength));
  47.         // Perform tests for strings five characters long
  48.     }
  49.  
  50.     @Theory
  51.     public void takesAnInteger(int dataToTest) {
  52.         assumeThat(dataToTest, is(not(0)));
  53.         // Perform tests for non-zero integers.
  54.     }
  55.  
  56.     @Theory
  57.     public void takesALong(long dataToTest) {
  58.         long lowerLimit = 20L;
  59.         long upperLimit = 98L;
  60.         assumeThat(dataToTest,
  61.             both(greaterThanOrEqualTo(lowerLimit)).and(lessThanOrEqualTo(upperLimit)));
  62.         // Perform tests needing a long in the [20, 98] range.
  63.     }
  64. }

When this example is run, Eclipse will generate the following test report:

Theory Run

The report fails the needsToContainW() test because none of the generated data matched the assumeThat matcher. This means that, while a failed assumeThat() will not fail a test, a Theories runner insists that a test is exercised at least once during the test run or it fails that test. Which makes sense and is a nice feature. This permits a tester to know when the data being supplied by the DataPoints is not exercising everything that needs testing.

While a Theories test class is not going to be as quick to write as a Parameters test class, there will be cases -- specifically when test data is generated, rather than known before hand -- when a Theory test will be a better fit than a Parameters test.

More information on Theories and the background behind them can be found at the JUnit wiki page, Theories.

Rules

Originally called Interceptors, Rules allow a tester to insert behavior before and after a test is run. JUnit comes with several very handy Rules. These can be used as is or extended. Or a tester can write their own.

Rules provided by JUnit include:

The Javadoc for each of the provided Rules has examples of their use. They all implement the TestRule interface and theTestRule Javadoc provides a handy jump point to the documentation for each of the individual rules. In addition, the JUnit Wiki Rules page also illustrates their use.

Rules provide a tester with tremendous control over the execution of tests. To give some idea of this, here's a quick example of writing your own Rule. Three things are necessary: a Test using the Rule, a Rule, and a Statement. The Javadoc for Statement describes the class as representing one or more actions to be taken at runtime. For the purposes of this example, you can think of that as being the test.

In practice, most testers implementing a Rule to control the behavior of their tests will start with a base of TestWatcher, which implements TestRule, rather than implementing the TestRule directly like the following example.

First the Test:

  1. package com.ociweb;
  2.  
  3. import org.junit.Rule;
  4. import org.junit.Test;
  5. import org.junit.rules.TestName;
  6.  
  7. public class ANewRuleTest {
  8.     @Rule
  9.     public ANewRule customRule = new ANewRule();
  10.  
  11.     @Rule
  12.     public TestName testName = new TestName();
  13.  
  14.     @Test
  15.     public void firstTestOfCustomRule() {
  16.         System.out.println("Test -> in the test named: " + 
  17.                 testName.getMethodName());
  18.     }
  19.  
  20.     @Test
  21.     public void secondTestOfCustomRule() {
  22.         System.out.println("Test -> in the test named: " + 
  23.                 testName.getMethodName());
  24.     }
  25. }

Now the Rule:

  1. package com.ociweb;
  2.  
  3. import org.junit.rules.TestRule;
  4. import org.junit.runner.Description;
  5. import org.junit.runners.model.Statement;
  6.  
  7. public class ANewRule implements TestRule {
  8.     @Override
  9.     public Statement apply(Statement theTest, Description description) {
  10.         System.out.println("Rule  -> ready to 'apply' the Statement for "
  11.                 + description);
  12.         return new ANewRuleStatement(theTest);
  13.     }
  14. }

Finally, the Statement:

  1. package com.ociweb;
  2.  
  3. import org.junit.runners.model.Statement;
  4.  
  5. public class ANewRuleStatement extends Statement {
  6.  
  7.     private final Statement theTest;
  8.  
  9.     public ANewRuleStatement(Statement theTest) {
  10.         this.theTest = theTest;
  11.     }
  12.  
  13.     @Override
  14.     public void evaluate() throws Throwable {
  15.         System.out.println("Stmt -> before the test");
  16.         try {
  17.             System.out.println("Stmt -> running the test");
  18.             theTest.evaluate();
  19.         } finally {
  20.             System.out.println("Stmt -> after the test");
  21.         }
  22.     }
  23. }

Here is the output from running the test:

    Rule -> ready to 'apply' the Statement for secondTestOfCustomRule(com.ociweb.ANewRuleTest)
    Stmt -> before the test
    Stmt -> running the test
    Test -> in the test named: secondTestOfCustomRule
    Stmt -> after the test
    Rule -> ready to 'apply' the Statement for firstTestOfCustomRule(com.ociweb.ANewRuleTest)
    Stmt -> before the test
    Stmt -> running the test
    Test -> in the test named: firstTestOfCustomRule
    Stmt -> after the test

If this sort of control would be useful, take a look at the TestWatcher Rule example on the Rules Wiki page and theTestWatcher Javadoc to see how it provides even more access to test events.

Order of Test Execution

One of the most common questions you encounter about using JUnit is "How do I make my tests run in a certain order?". Since JUnit uses reflection to determine which tests are to be run, the answer until recently is, "You can't.".

Until now.

Starting wth version 4.11, JUnit provides the ability to fix the order of method executions. By annotating your class with the @FixMethodOrder annotation, you can determine the order of the tests. You do this by decorating the test class with@FixMethodOrder(MethodSorters.NAME_ASCENDING) and adopting a naming scheme for your tests that will place them in the order you want the tests to execute in. Please note the names are sorted in lexicographic order to determine the execution order so if you are using numbers in the names, remember that 12 comes before 2, but not before 02. Details on using the @FixMethodOrder are at Text execution order.

Summary

The JUnit team has done an excellent job equipping their users with fine tools. Now that you've reviewed a few, pick one that seems useful to your purposes and put it into play this week. Then try another the week after that. As with most productive tools, before long you'll wonder how you ever lived with just assert.

But please don't quit using asserts.

Annotated References