Rolf Engelhard

Fortunately, the team I’m currently working in emphasizes on a high degree of test coverage. First we started with JUnit4 but switched to TestNG after a while concerning design and performance purposes. Although we migrated most of the existing tests, we stuck in the situation that there were still JUnit4- and JUnit3-tests which couldn’t be converted to TestNG. Those tests depend on third-party-libraries which in turn are related to JUnit.

So how to execute TestNG, JUnit3- and JUnit4-tests on the same compilation run using Maven?

Imagine a project with just one TestNG-, one JUnit3- and one JUnit4-test. Simply adding the Surefire plugin to the pom.xml (and of course the JUnit4- and TestNG-dependencies) will make Surefire use TestNG as default. Thus a call of mvn clear package will lead to:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.194 sec

As you can see, Surefire triggers just two of the tests (TestNG and JUnit3). That’s because:

• TestNG executes TestNG- and JUnit3-tests only
• JUnit4 executes JUnit3- and JUnit4-tests only

See [1] for details.

Solution 1: Profiles

So the first solution for this issue bases on profiles. Beside the default Surefire-run with TestNG there is an additional profile (here named “JUnitRun”) in which the attribute “TestNGArtifactName” is set to “none”. As a result JUnit4 will be taken by Surefire when calling Maven with the profile “JUnitRun”.

<profiles>
      <profile>
          <id>JUnitRun</id>
          <activation>
              <activeByDefault>false</activeByDefault>
          </activation>
          <build>
              <plugins>
                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-surefire-plugin</artifactId>
                      <configuration>
                      <testNGArtifactName>none:none</testNGArtifactName>
                      </configuration>
                  </plugin>
              </plugins>
          </build>
      </profile>
</profiles>
<build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-surefire-plugin</artifactId>
          </plugin>
      </plugins>
</build>

Although this provides a solution it also has a big drawback: you need two compilations to get all tests executed (mvn clean install and mvn clean install -P JunitRun). In addition the JUnit3-tests are executed two times (once a run). This isn’t acceptable for a continuous integration system. So how to join two executions to one?

Solution 2: Two executions in one run

Maven can trigger more than just one execution of the same plugin [2]. This is utilized in the following solution: there’s a (default) execution for running TestNG and a second execution which TestNGArtifactName is set to “none” (again for calling JUnit4 instead of TestNG). So the pom.xml looks like this:

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <executions>
            <execution>
                <phase>test</phase>
                <goals>
                    <goal>test</goal>
                </goals>
                <configuration>
                    <testNGArtifactName>none:none</testNGArtifactName>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

And the command line shows that all tests have been executed:

Running TestSuite
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.19 sec
[...]
Running JUnit3Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec
Running JUnit4Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec

But the drawback remains that the JUnit3-tests are triggered two times. In a project with many JUnit3-tests and/or some long running ones this is an indefensible circumstance. Also reporting tools might get confused while calculating their results.

Solution 3: Two executions, one restricted

Improving the second attempt can be done by deciding whether the TestNG- or the JUnit4-tests should be treated in a special way. Assuming that there are just a few JUnit4- but lots of TestNG-tests, the final solution is to call every TestNG- and every JUnit3-test within the default execution (as it is already done in solution 2). But now the automatic execution of tests in the second run is rejected. Instead the second execution triggers the JUnit4-tests explicitly by using the “includes”-tag. Inside those tags it is considerable to either list up the JUnit4-tests one by one or follow a team-agreement like “every Junit4-Test ends with *Junit4Test”—as it is done in the following listing:

<build>
    <plugins>
        <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <executions>
            <execution>
                <phase>test</phase>
                <goals>
                <goal>test</goal>
                </goals>
                <configuration>
                    <testNGArtifactName>none:none</testNGArtifactName>
                    <reportsDirectory>${project.build.directory}/surefire-reports/junit-junit4-results</reportsDirectory>
                    <includes>
                        <include>**/*JUnit4Test.java</include>
                    </includes>
                </configuration>
            </execution>
        </executions>
        </plugin>
    </plugins>
</build>

With these changes Maven will print to the command line:

Running TestSuite
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.19 sec
[...]
Running JUnit4Test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec

As expected all TestNG, JUnit3- and JUnit4-tests will have been executed—and every one only once. Happy testing.