JMLUnitNG Usage

This documentation should allow you to use JMLUnitNG to generate and run unit tests; it is not exhaustive, but will be augmented over time. API documentation is also available.

Command Line Options

JMLUnitNG is invoked as java -jar jmlunitng.jar [OPTIONS] path-list where path-list is a space-separated list of filesystem paths (to files or directories) and [OPTIONS] is 0 or more of the following command line options. All Java files in the specified paths, recursively, except for those generated by JMLUnitNG are processed according to the command-line options.
  • -d, --dest [DIRECTORY] : Use DIRECTORY as the output directory for generated classes. If this is not specified, test classes are generated in the same directory as the classes under test.
  • -cp , --classpath : Use the given list of directories and Jar files (formatted as for javac) as the classpath during parsing. By default, the CLASSPATH environment variable is used.
  • -sp , --specspath : Use the given list of directories and Jar files (formatted as for javac) as the specspath during parsing. By default, the SPECSPATH environment variable is used.
  • --rac-version : Generate RAC handling code for the specified JML RAC version. The default is ‘openjml’ for OpenJML RAC. The other supported values are 'jml2' and 'jml4' for JML2 and JML4 RAC (respectively).
  • --deprecation : Generate tests for deprecated methods. Deprecated methods are detected using the @Deprecated Java annotation (and not, in the current version of JMLUnitNG, by the @deprecated Javadoc tag). This option is off by default, which means deprecated methods will not be tested.
  • --inherited : Generate tests for inherited methods. This option is off by default, which means that tests will not be generated for methods whose bodies are inherited from parent classes. Turning it on causes tests to be generated for all (concrete) methods in every class under test.
  • --public : Generate tests only for public methods. This is the default behavior of JMLUnitNG.
  • --protected : Generate tests for protected and public methods.
  • --package : Generate tests for package (no protection modifier), protected and public methods.
  • --parallel : Generate data providers that default to running in parallel. This option is off by default, which means that multiple tests of the same method will run sequentially (regardless of your TestNG settings); turning it on enables them to run in parallel with appropriate TestNG settings.
  • --reflection : Generate test data reflectively. This option is off by default, which means that no objects will be automatically generated on which to call test methods; turning it on causes such objects to be generated.
  • --children : For all parameters, generate test data using not only the parameter class but also any children of that class that are explored when generating the tests. This allows many methods that take interface/abstract class parameters to be tested automatically.
  • --literals : Use literals found in classes and methods as default data values for testing those classes and methods (literals found outside methods, e.g. in static fields, are used for all methods).
  • --spec-literals : Use literals found in class and method specifications as default data values for testing those classes and methods (literals found in class specifications are used for all methods).
  • --clean : Remove from the destination path all old JMLUnitNG-generated files, including any manual modifications. If no destination path is set, all files and directories in path-list are cleaned. This option should be used with care, as it removes all JMLUnitNG-generated files, recursively, in the destination path or path-list, regardless of when/how they were generated.
  • --prune : Remove from the destination path any old JMLUnitNG-generated files for path-list that do not conform to the current API of the classes under test and the current JMLUnitNG options. If no destination path is set, all files and directories in path-list are pruned. This option should be used with care, as it removes every JMLUnitNG-generated file, recursively, in the destination path or path-list, that would not have been created in the current JMLUnitNG run. It is generally used when you change the API of a class under test but want to preserve the old customized data strategies for the method parameters that have not changed; a --clean would delete all the old strategies, while a --prune only deletes the ones that are no longer useful.
  • --no-gen : Do not generate tests. This option is generally used in conjunction with --clean or --prune to remove unwanted JMLUnitNG-generated files without generating new ones.
  • --dry-run : Display status/progress information about the operations that would be performed but do not modify the filesystem. When used with any other set of options, --dry-run causes JMLUnitNG to run through the entire test generation process and display the steps, but to generate no output; it is useful for seeing what files would be deleted with a --clean or --prune, or what methods would have tests generated for them with specific sets of options.
  • -v, --verbose : Display status/progress information.
  • -h, --help : Display a list of command line options with abbreviated documentation.
Typical Usage

In general, testing a class (or set of classes) with JMLUnitNG involves the following steps:
  1. Generate the test classes, by running JMLUnitNG (java -jar jmlunitng.jar) with appropriate command line options. It is strongly recommended that this step be performed with a non-Apple (i.e., OpenJDK 7) virtual machine on Mac OS X; see the OpenJML on Mac OS X page for details.
  2. (Optionally) Modify the test data generation strategies to add customized test data (for information about the generated files themselves, see below). If you do not change the test data generation strategies, they will use default data. The primitive type data defaults are the same as those for JMLUnit (e.g., -1/0/1 for int); the non-primitive type data default is null and, if reflective test data generation (--reflective) is turned on, non-primitive data objects and arrays are also generated based on the primitive type strategies.
  3. Compile the classes under test with the appropriate JML compiler (openjml, jmlc, and jml4c for OpenJML, JML2, and JML4, respectively). Do not compile the generated classes with the JML compiler! (they will warn you at runtime if you do)
  4. Compile the generated classes with a regular Java compiler, with the appropriate JML runtime Jar and the JMLUnitNG Jar in the CLASSPATH. If you are using OpenJML RAC you only need the JMLUnitNG Jar in the CLASSPATH, as the JMLUnitNG Jar includes OpenJML.
  5. Run the tests. This can be done by writing a testng.xml file to run all your tests, by running the test classes individually from the command line (each test class has a main method), or by running TestNG from the command line and pointing it at the test classes. Regardless of the method you choose to run the tests, you must have the JMLUnitNG Jar and, for JML2 or JML4, the appropriate JML runtime Jar, in the CLASSPATH.
Generated Files

JMLUnitNG generates several Java source files for each class under test. Five different kinds of Java class are generated: the test class, which contains all the TestNG tests; package-level strategy classes for method parameter types, which allow you to specify data values to be used everywhere a given type is used in the package containing the class under test; class-level strategy classes for method parameter types, which allow you to specify data values to be used everywhere a given type is used in the class under test; an instance strategy class for the class under test, which allows you to specify objects on which to call test methods; and method parameter strategy classes for every parameter of every method under test, which allow you to specify data values for individual method parameters.

For example, assume that you generate tests for one class named ClassName, which is in package P and has the following 2 methods:
  • public void m(int a, String b)
  • public void m(long a, String b)
In this case, JMLUnitNG will generate the following 12 Java source files:
  • ClassName_JML_Test.java - the class that contains all the generated TestNG tests. This class has a main method to run the tests, and can also be used with TestNG in any method supported for running TestNG tests.
  • ClassName_InstanceStrategy.java - the strategy class for generating instances of class ClassName. If reflection is turned on, this will generate such instances by invoking the ClassName constructors with automatically-generated data values; in any case, it will always try to generate an instance using the empty constructor. Instances of ClassName to be tested can also be added to this class explicitly.
  • PackageStrategy_int.java - the package-level data strategy for type int. int values added to this file will be used as input for every int parameter of every method of every class in package P (in this case, there is only one such class).
  • PackageStrategy_java_lang_String.java - the package-level data strategy for type java.lang.String. String values added to this file will be used as input for every String parameter of every method of every class in package P (in this case, there is only one such class).
  • PackageStrategy_long.java - the package-level data strategy for type long. long values added to this file will be used as input for every long parameter of every method of every class in package P (in this case, there is only one such class).
  • ClassStrategy_int.java - the class-level data strategy for type int. int values added to this file will be used as input for every int parameter of every method of ClassName (in this case, there is only one such parameter).
  • ClassStrategy_java_lang_String.java - the class-level data strategy for type java.lang.String. String values added to this file will be used as input for every String parameter of every method of ClassName (in this case, there are two such parameters).
  • ClassStrategy_long.java - the class-level data strategy for type long. long values added to this file will be used as input for every long parameter of every method of ClassName (in this case, there is only one such parameter).
  • m__int_a__String_b__a.java - the strategy class for generating data to use for parameter a of method m with signature (int a, java.lang.String b). int values added to this file will be used only for this specific parameter of this specific method.
  • m__int_a__String_b__b.java - the strategy class for generating data to use for parameter b of method m with signature (int a, java.lang.String b).
  • m__long_a__String_b__a.java - the strategy class for generating data to use for parameter a of method m with signature (long a, java.lang.String b).
  • m__long_a__String_b__b.java - the strategy class for generating data to use for parameter b of method m with signature (long a, java.lang.String b).
All files, except the TestNG test class, the instance strategy, and the package strategies, are generated in their own package - P.ClassName_JML_Data - to avoid cluttering the package of the class under test. If the class under test is not package, all the generated classes are generated with no package (and their names are slightly different than described above for disambiguation purposes). Also note that the local strategy filenames (with method signatures) will have a number in them to help disambiguate signatures containing identically named classes from different packages. The exact format of these filenames is subject to change in future versions of JMLUnitNG.

Note that it is always possible to override the data values for any strategy higher in the inheritance hierarchy. For instance, if for a particular method parameter you want reflection turned off and to use only some specific data values, but reflection is turned on at the package level and a number of other data values have been added there, you can override the relevant methods of the strategy (e.g., packageValues) to return empty data sets and call the relevant control methods (e.g., setReflective) to modify the strategy’s behavior. See the Javadoc for Strategy and NonPrimitiveStrategy for more details.