Running TestNG-, JUnit3- and JUnit4-tests with Maven Surefire in one run

Fortunately, the team I’m currently working in emphasizes on a high degree of test coverage. First we started with JUnit4 but switched to TestNG after a while concerning design and performance purposes. Although we migrated most of the existing tests, we stuck in the situation that there were still JUnit4- and JUnit3-tests which couldn’t be converted to TestNG. Those tests depend on third-party-libraries which in turn are related to JUnit.

So how to execute TestNG, JUnit3- and JUnit4-tests on the same compilation run using Maven?

Imagine a project with just one TestNG-, one JUnit3- and one JUnit4-test. Simply adding the Surefire plugin to the pom.xml (and of course the JUnit4- and TestNG-dependencies) will make Surefire use TestNG as default. Thus a call of mvn clear package will lead to:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.194 sec

As you can see, Surefire triggers just two of the tests (TestNG and JUnit3). That’s because:

  • TestNG executes TestNG- and JUnit3-tests only
  • JUnit4 executes JUnit3- and JUnit4-tests only

See [1] for details.

Solution 1: Profiles

So the first solution for this issue bases on profiles. Beside the default Surefire-run with TestNG there is an additional profile (here named “JUnitRun”) in which the attribute “TestNGArtifactName” is set to “none”. As a result JUnit4 will be taken by Surefire when calling Maven with the profile “JUnitRun”.

<profiles>
      <profile>
          <id>JUnitRun</id>
          <activation>
              <activeByDefault>false</activeByDefault>
          </activation>
          <build>
              <plugins>
                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-surefire-plugin</artifactId>
                      <configuration>
                      <testNGArtifactName>none:none</testNGArtifactName>
                      </configuration>
                  </plugin>
              </plugins>
          </build>
      </profile>
</profiles>
<build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-surefire-plugin</artifactId>
          </plugin>
      </plugins>
</build>

Although this provides a solution it also has a big drawback: you need two compilations to get all tests executed (mvn clean install and mvn clean install -P JunitRun). In addition the JUnit3-tests are executed two times (once a run). This isn’t acceptable for a continuous integration system. So how to join two executions to one?

Solution 2: Two executions in one run

Maven can trigger more than just one execution of the same plugin [2]. This is utilized in the following solution: there’s a (default) execution for running TestNG and a second execution which TestNGArtifactName is set to “none” (again for calling JUnit4 instead of TestNG). So the pom.xml looks like this:

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <executions>
            <execution>
                <phase>test</phase>
                <goals>
                    <goal>test</goal>
                </goals>
                <configuration>
                    <testNGArtifactName>none:none</testNGArtifactName>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

And the command line shows that all tests have been executed:

Running TestSuite
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.19 sec
[...]
Running JUnit3Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec
Running JUnit4Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec

But the drawback remains that the JUnit3-tests are triggered two times. In a project with many JUnit3-tests and/or some long running ones this is an indefensible circumstance. Also reporting tools might get confused while calculating their results.

Solution 3: Two executions, one restricted

Improving the second attempt can be done by deciding whether the TestNG- or the JUnit4-tests should be treated in a special way. Assuming that there are just a few JUnit4- but lots of TestNG-tests, the final solution is to call every TestNG- and every JUnit3-test within the default execution (as it is already done in solution 2). But now the automatic execution of tests in the second run is rejected. Instead the second execution triggers the JUnit4-tests explicitly by using the “includes”-tag. Inside those tags it is considerable to either list up the JUnit4-tests one by one or follow a team-agreement like “every Junit4-Test ends with *Junit4Test”—as it is done in the following listing:

<build>
    <plugins>
        <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <executions>
            <execution>
                <phase>test</phase>
                <goals>
                <goal>test</goal>
                </goals>
                <configuration>
                    <testNGArtifactName>none:none</testNGArtifactName>
                    <reportsDirectory>${project.build.directory}/surefire-reports/junit-junit4-results</reportsDirectory>
                    <includes>
                        <include>**/*JUnit4Test.java</include>
                    </includes>
                </configuration>
            </execution>
        </executions>
        </plugin>
    </plugins>
</build>

With these changes Maven will print to the command line:

Running TestSuite
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.19 sec
[...]
Running JUnit4Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec

As expected all TestNG, JUnit3- and JUnit4-tests will have been executed—and every one only once. Happy testing.

Reference

  1. [1] http://groups.google.com/group/testng-users/msg/b1fa90f8877134c8
  2. [2] http://maven.apache.org/guides/mini/guide-configuring-plugins.html#Using_the_executions_Tag

Avoid creation of emailable-report.html when using Maven, Surefire and TestNG

The problem

Last week, my Eclipse ran into a lot of “java.lang.OutOfMemoryError: Java heap space”-errors, telling me about that incidence (using a pop-up) and asking me if I really want to work on with my workspace (using an additional pop-up) again and again. Coding became hardly practicable so I started to search the cause of this failure.

What was the problem? The html-validation plugin of Eclipse tried to scan a file called “emailable-report.html” inside the target-directory (more precisely: in the directory target/surefire-reports). That directory had been created by m2eclipse but a simple “mvn clean install” created it, too.

Having a closer look on emailable-report.html with my favorite file manager I couldn’t believe my eyes: that file had a size of 440MB! I also found an emailable-report.html in every projects target folder inside my workspace, some smaller, some larger. The largest: 1.4GB—and by the way—who sends reports with that size via email?? I opened one of these files with a text editor and found the test results of the TestNG-tests, which are executed on a Maven build. Depending on a large set of test data (here: thousand of useragents) those files were blown up with information.

So the html-validation plugin simply crashed because of a (very) large html-file.

The solution

At first I blamed the Maven Surefire plugin for creating the reports but couldn’t find an option to turn off this behavior. In fact I even didn’t find a relation between Surefire and the emailable-report.html. And that’s the point–those files aren’t written by Surefire, they are written by TestNG itself! After googling a while I got to the property “usedefaultlisteners”. Unfortunately, it isn’t specified in the main TestNG-manual but in the TestNG-Ant manual. However, negating this option will shut down the result-writing (as well as all other default listeners).

So I added the following to the Surefire configuration in my pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <properties>
            <property>
                <name>usedefaultlisteners</name>
                <key>false</key>
            </property>
        </properties>
    </configuration>
</plugin>

After a “mvn clean install” the problem was gone. There is also a nice side-effect of this modification: before editing the maven build took about 2:30min on my developer-machine, now it’s done in 1:30min.