Table of Contents
- Table of Contents
- Various Test Techniques
- HTML scraping
- Submitting forms
- Configuration round-trip testing
- Web page assertions
- TestCase as a RootAction
- Testing authorization behaviours
- Custom builder
- Registering Extensions during tests
- Problems and hacks for running tests in Windows
- Test harness annotations
- Informational annotations
- Test environment annotations
Jenkins comes with a test harness built around JUnit to make test development simpler. This harness provides the following features:
- Starts an embedded servlet container so that the test code can exercise user interaction through HTTP and assert based on the outcome.
- HtmlUnit with a bit of enhancements allows you to productively test HTTP/UI interaction.
- Prepares and tears down a fresh Jenkins instance for each test case. So each test method will run in a fresh environment, isolated from other tests.
- Test code can also directly access Jenkins object model. This allows tests to assert directly against the internal state of Jenkins, as well as perform some operations directly, as opposed to doing so only through the HTML scraping.
- Declarative annotations to specify the environment in which a test will be run. For example, your test method can request that Jenkins shall be started with a certain
- Declarative annotations to maintain association between tests and bugs/discussions.
If you are using the Plugin Parent POM 2.3 or later the version of the Jenkins Test Harness (which starting in version 2.0 is an artifact independent from Jenkins Core) to be used for the tests can be configured using the
jenkins-test-harness.version. For 1.x versions of the Plugin POM the version used is the same of the parent POM which is equal to the Jenkins baseline version. The recommended version of Jenkins Test Harness is 2.1 or above unless your plugin depends on Jenkins < 1.580.
The following code shows a very simple test case. Your test will use JenkinsRule to provide test fixtures for some of those features outlined above.
Each test method will start with a fresh temporary installation of Jenkins. The
first method doesn't request any particular Jenkins data set to seed Jenkins, so it will start from an empty installation.
The test then proceeds to create a new project and set it up. As you can see, the code directly talks to the in-memory Jenkins object model. (There is the
jenkins variable defined in
JenkinsRule that you can use for access.) While you can do the same by emulating the user HTTP interaction through HtmlUnit, this way is often a convenient way to prepare an environment for the code that you want to test.
The test code in this example then switch to HtmlUnit to emulate the UI interaction and verify that we get results that we expect.
When a test completes, the temporary Jenkins installation will be destroyed.
You can use
@ClassRule to speed up the test suite if you are positive that the test cases will not interfere with each other.
How to set environment variables
When preparing your virtual test environment, you may wish to simulate Jenkins environment variables that can be set on the Jenkins configuration page. Adding environment variables to a Jenkins instance before a test is simple, as the example below demonstrates.
Your IDE will most likely have the ability to select a single JUnit test and execute it in the debugger. Otherwise you can run
mvn -Dmaven.surefire.debug -Dtest=hudson.SomeTest test to accomplish this.
To debug slaves launched by Jenkins, set
-Dorg.jvnet.hudson.test.HudsonTestCase.slaveDebugPort=PORT to the system property, or from your test case set
JenkinsRule.SLAVE_DEBUG_PORT to a non-0 value.
Various Test Techniques
Sometimes you want to have quick tests which don't start up a 'full' Jenkins instance - as
JenkinsRule does - as this can take some time. In that case you shouldn't have your test classes use
JenkinsRule. (Or you can use
@WithoutJenkins on certain methods.)
As creating most Jenkins core classes without a Jenkins instance is unfortunately not easy, stubbing can come in handy. One excellent stubbing and mocking framework is e.g. Mockito. For example, if you want to stub a build with a certain result you could do:
See Mocking in Unit Tests for more info.
If you'd like to test the HTML generated by Jenkins, XPath test is often convenient.
So you'd have to look for the corresponding HtmlButton element, then use that to call the submit method, like this:
The original HtmlUnit doesn't really do a good job of chaining all exceptions together, so we are patching HtmlUnit to make sure it retains the full stack trace leading up to the root cause. If you found a case where this chain is broken, please file a bug.
Configuration round-trip testing
- Programmatically construct a fully populated instance
- Request a configuration page via HtmlUnit
- Submit the config page without making any changes
- Verify that you still have the identically configured instance
This test ensures that your configuration page is properly pre-populated with the current setting of your model object, and it also makes sure that the submitted values are correctly reflected on the constructed model object. To be really sure, do this twice with different actual values — for example, you should try a non-null string and null string, true and false, etc., to exhaust representative cases.
Web page assertions
HtmlUnit has a
WebAssert class that can be used for simple assertions on HTML pages.
To assert that the System configuration page contains the CVS SCM configuration entry:
To find Elements by name= vs. id=, use
An example from the ironmqNotifier plugin.
See note on pom.xml below if this example gives you
When changing from 1.625.1 to 1.625.2, there appears to be a difference in the support library for HtmlPage.
When parsing pages for Number (Long or Integer), you may get the following error:
The supporting library appears to need
HtmlNumberInput instead of
HtmlTextInput to function correctly with numbers.
Make the following change...
Note; If you wish to move back to a version prior to 1.625.2, you will need to modify your Tests as the previous library associated with HtmlPage will not recognize HtmlNumberInput as part of the class.
isRunAsTest variable defined in
hudson-behavior.js, which is included in all the pages. This can be used to disable some ajax operations, for example. Obviously, this has to be used with caution so that tests will continue to test the real thing as much as possible.
TestCase as a RootAction
An instance of the test case being executed is added to Jenkins' URL space as
JenkinsRule is itself a
RootAction. Among other things, this enables your test class to define Jelly views, and invoke it like
Testing authorization behaviours
Unit test harness contains a SecurityRealm implementation suitable for unit tests. This can be installed to as follows:
This virtual security realm allows login attempts by any user name so long as its password is exactly the same as the user name.
WebClient.login method provides a convenient method that allows you to login a session object.
You can extend TestBuilder to write a one-off builder that can coordinate with your test. This is often convenient to stage things up for testing your Publisher, for example by placing files in the workspace, etc.
OneShotEvent is also often an useful companion so that the thread that runs your test method and the thread that runs the build can coordinate — for example, the following program blocks the main thread until the build starts.
Registering Extensions during tests
During the test, one might want to register extensions just during that particular test, for example to assist the test scenario. You can do this by defining such extension as a nested type of your test case class and put TestExtension instead of Extension.
It lets you tie an extension to just one test method, or all test methods on the same class.
Problems and hacks for running tests in Windows
See Unit Test on Windows.
Test harness annotations
There are several annotations in the Jenkins test framework.
Related issue id in tracker.
Production classes that tests are related to. Useful when the relationship between the test class name and the test target class is not obvious.
URL to the web page indicating a problem related to this test case.
URL to the e-mail archive. Look for the e-mail in http://jenkins.361315.n4.nabble.com/Jenkins-users-f361316.html or http://jenkins.361315.n4.nabble.com/Jenkins-dev-f387835.html
Test environment annotations
The specified class will be used to set up the test environment using
Runs a test case with a data set local to test method or the test class.
This recipe allows your test case to start with the preset
HUDSON_HOME data loaded either from your test method or from the test class.
For example, if the test method is
org.acme.FooTest.bar(), then you can have your test data in one of the following places in resources folder (typically
org/acme/FooTest/bardirectory (that is, you'll have
org/acme/FooTest/bar/config.xml), in the same layout as in the real
org/acme/FooTest/bar.zipas a zip file.
org/acme/FooTestdirectory (that is, you'll have
org/acme/FooTest/config.xml), in the same layout as in the real
org/acme/FooTest.zipas a zip file.
Search is performed in this specific order. The fall back mechanism allows you to write one test class that interacts with different aspects of the same data set, by associating the dataset with a test class, or have a data set local to a specific test method.
The choice of zip and directory depends on the nature of the test data, as well as the size of it.
Runs a test case with one of the preset
HUDSON_HOME data set:
NO_ANONYMOUS_READACCESS- Secured Jenkins that has no anonymous read access. Any logged in user can do anything.
ANONYMOUS_READONLY- Secured Jenkins where anonymous user is read-only, and any logged in user has a full access.
Installs the specified plugin before launching Jenkins in the test. For now, this has to be one of the plugins statically available in resources
Runs a test case without create and tear down a Jenkins instance.
This Recipe has to be used in test methods that do not use a 'full' Jenkins instance, but are part of a test class that uses
HudsonTestCase to prevent performance issues.
Runs a test case with the given timeout expressed in seconds.
HudsonTestCase (JUnit 3)
HudsonTestCase is available for JUnit 3 tests. The functionality is similar to
When seeing the following error : java.lang.NoClassDefFoundError: org/hamcrest/MatcherAssert when running mvn:test if you are using WebPage Assertions per the above example.
It may be necessary to add org.hamcrest / hamcrest-all to your pom.xml file to avoid a matcher error. The cause is not yet known but this workaround will allow you to continue to learn how to create appropriate tests.
- Host test harness javadoc and link from this article.
HtmlPagesubclass to define more convenience methods.HtmlNumberInput inputElement = page.getElementByName("_.defaultExpirySeconds");