Plugin Information |
---|
View TAP on the plugin site for more information. |
Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:
TAP Plug-in depends on tap4j - a TAP implementation for Java, and on the JUnit Plug-in. You may get errors if the JUnit Plug-in is not active in your Jenkins instance (see JENKINS-27227 for more).
Overview
The plug-in looks for TAP files like the following one.
1..2 ok 1 - Yahoo! not ok 2 - org.tap4j.Error...
When a TAP stream like the above is found, the plug-in delegates the parsing the tap4j. The results of tap4j parsing are then analysed, organized and displayed to the user as graphs and custom pages.
You can test your TAP streams with tap4j, using InstantTAP.
Understanding TAP streams in 2 minutes
One of the easiest ways to learn something new is by examples. So here we will show some examples of TAP streams. For being human-friendly, it shouldn't confuse you. We will use comments to explain each line.
1..2 # a plan stating that we have two tests cases, from 1 to 2. ok 1 - Yahoo! # the first test result was executed successfully and has a description of ' - Yahoo!'. not ok 2 - org.tap4j.Error... # unfortunately, the second test result failed. The description here was used to display some nasty Exception.
Running Perl tests with prove
The plug-in cannot handle prove's default output (since it includes more information than simply TAP, and causes tap4j parser to fail). The best way to handle the prove output is by using Perl module TAP::Harness::Archive. Supposing you have your tests under t/ directory, you can create another directory (say, output) and archive your TAP tests with prove by using a command line similar to the below:
prove t/ --archive output
The result files will be stored under output/t/. You can use a pattern in the plug-in configuration like t/*/.t.
Using attachments
The following is a TAP with attachments, using YAMLish. If you are familiar with YAML, this example should be very easy to read.
1..2 ok 1 not ok 2 - br.eti.kinoshita.selenium.TestListVeterinarians#testGoogle --- extensions: Files: my_message.txt: File-Title: my_message.txt File-Description: Sample message File-Size: 31 File-Name: message.txt File-Content: TuNvIGNvbnRhdmFtIGNvbSBtaW5oYSBhc3T6Y2lhIQ== File-Type: image/png ...
The Files entry has an array of files. You have to Base64 encode your content data.
Subtests (grouping or test suites).
Subtests let you group several TAP streams unto a single one. This way, you can organize your tests in similar fashion to JUnit or TestNG test suites. Indentation is important for TAP subtests.
Subtests and YAMLish are not officially in TAP 13 specification
1..3 ok 1 - First test 1..2 ok 1 - This is a subtest ok 2 - So is this 1..2 ok 1 - This is a subtest ok 2 - So is this ok 2 - An example subtest ok 3 - Third test
Configuration
- Install the Jenkins TAP Plug-in using the Plug-in Manager or manually by copying the .hpi (or .jpi) file
- Check the option to publish TAP, configure a pattern (and other settings)
- Execute your build and analyze the results
Screenshots
Jenkins JUnit compatible reports and graphs
Custom actions for TAP too
YAMLish support
Languages Supported
- English (American)
- Portuguese (Brazil)) Work in Progress
- Spanish (Thanks to César Fernandes de Almeida) Work in Progress
Known Limitations
- If the file type of a TAP report is considered as binary by the Jenkins webserver then the TAP plugin doesn't consider this file for inclusion in TAP reports (see #15813 for further details). To make sure your TAP report is considered for inclusion use e.g. the file name suffix .tap (so instead of a file named "report" use "report.tap").
Compatibility Matrix
This plug-in is not developed for Hudson, and we don't fix bugs happening in any version of Hudson.
Plugin | 1.0 | 1.1 | 1.2.x | 1.3~1.9 | 1.10 | 1.11 | 1.12 | 1.13 | 1.14 | 1.15 | 1.16 | 1.17 | 1.18+ |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
TAP | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 |
Jenkins | 1.399+ | 1.399+ | 1.399+ | 1.399+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ | 1.488+ |
Hudson | ? | ? | ? | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported | Not supported |
Open Issues
Sponsors
For commercial support, please get contact us via @tupilabs
Resources
- This plug-in is going to be part of "Make your tests speak TAP" presentation in JCertif by Bruno P. Kinoshita, in September 2011. In this presentation will be presented too the tap4j project, how to enable TAP in JUnit and TestNG, integrate Perl and Java tests and an Eclipse TAP editor.
- Gábor Szabó (2009), Test Reporting system: Smolder wish-list <http://szabgab.com/blog/2009/07/test-reporting-system-smolder-wish-list.html>.
- Gábor Szabó (2009), Reporting Test Results <http://szabgab.com/blog/2009/04/reporting-test-results.html>
- tap4j - The TAP implementation for Java.
- Test Anything Protocol (official webpage).
- Performance tests with phantomjs and yslow (uses the plug-in for plotting the TAP results)
- TAP Plugin for Matlab
JCertif 2011
JCertif - Make your tests speak TAP - Speaker: Bruno P. Kinoshita
September, 2011 - Brazzaville, Congo
Release Notes
2.3 (2019-06-05)
- tap4j dependency upgraded to 4.4.2 version
- enabled parsing of TAP documents with corrupted YAML content
- New option in advanced, to remove corrupted YAML elements (should work after upgrading the plug-in, but remember to back up first anyway!)
2.2.2 (2019-02-14)
- JENKINS-55787 - Switch labels from entry to checkbox In Progress
2.2.1 (2018-02-07)
- JENKINS-48925 - TAP plugin is not compatible with JEP-200 in Jenkins 2.102 Resolved
2.2 (2018-01-26)
- JENKINS-48925 - TAP plugin is not compatible with JEP-200 in Jenkins 2.102 Resolved
2.1 (2017-02-28)
- JENKINS-22250: "fail the build if no test results are present" option doesn't fail the job
- JENKINS-21456: Plugin runs despite earlier errors
- Upgraded tap4j dependency from 4.2.0 to 4.2.1 (latest)
2.0 (2016-08-20)
- JENKINS-29649: don't overwrite existing junit-results
- JENKINS-34000: Workflow/Pipeline Support for TAP Plugin
- JENKINS-33779: TAP plugin: enable detailed output for TAP with subtests
1.25 (2016-06-21)
- JENKINS-23851: Expand env vars in the ant pattern field
- JENKINS-17804: Add option to show only failures
- JENKINS-29650: Don't use a build step synchronization, defaulting to NONE.
- JENKINS-30751: Use same logic everywhere to say whether a test is a failure or not
- JENKINS-24843: Copy duration from Tabulated Tests
- pull request #6: fix duration_ms (thanks to @forrest79)
- JENKINS-22016 via pull request #10: Expand env vars in the ant pattern field (thanks to @evandy0)
- SECURITY-85: [tap] Can read all files on file system
1.24 (2015-10-01)
- JENKINS-19471: TAP parser does not conform to InstantTAP website
- JENKINS-24505: Tests are not marked as skipped with TAP plugin
- JENKINS-19676: TAP test description does not get escaped
- JENKINS-27227: TAP plugin fails to load with java.lang.NoClassDefFoundError: hudson/tasks/test/AbstractTestResultAction
- JENKINS-29153: tap plugin fails job when not have to
1.23 (2015-05-23)
1.22 (2015-03-07)
1.21
- Dropped
Version 1.20
Version 1.18
- Updated to tap4j-4.0.8
- JENKINS-22047: Add option to reduce noise in logs
- JENKINS-22036: NullPointer when there is no Test Plan
- JENKINS-17960: Indicate if tests don't go to plan
- JENKINS-21917: TAP results graph causes null pointer exception
- https://github.com/jenkinsci/tap-plugin/pull/4
Version 1.17
- JENKINS-22047: Add option to reduce noise in logs
- JENKINS-22036: NullPointer when there is no Test Plan
- JENKINS-17960: Indicate if tests don't go to plan
- JENKINS-21917: TAP results graph causes null pointer exception
- Pull request #4
Version 1.17
Version 1.16
- Updated to tap4j-4.0.5 (better subtests handling)
Version 1.15
Version 1.14
- Security bug reported by Kees J. via e-mail. This issue is related to exposing files that the user running Jenkins has access via the plug-in.
Version 1.13
Version 1.12
- JENKINS-18885: Parse errors with Git's TAP test suite, part 2
- JENKINS-17878: HTML test output in tapResults not escaped
- JENKINS-17855: TAP Stream results summary page contains links that fail
- JENKINS-17504: TAP Plugin generates bad detail links on "tapTestReport" page
Version 1.11
- JENKINS-17859: TAP report table show failed test message on ALL tests after the failed one.
- JENKINS-17855: TAP Stream results summary page contains links that fail
- JENKINS-17941: Parse errors with Git's TAP test suite
- JENKINS-17947: Nested TAP not parsed correctly
- JENKINS-17504: TAP Plugin generates bad detail links on "tapTestReport" page
Version 1.10
Version 1.9
Version 1.8
- JENKINS-15914: TAP results table misses first comment line
- JENKINS-15322: NOTESTS in TAP response gives parse error and stack trace from plugin
- JENKINS-15401: support TODO directive to not fail such tests
- JENKINS-15907: When multiple TAP files with same basename match pattern, only one is processed
Version 1.7
Version 1.6
- JENKINS-15419: TAP published results hide JUnit published results
- JENKINS-15497: Display link to download TAP attachment
Version 1.2.7
- Removed requirement to have the TAP Plan at start or at the end of the TAP Stream. This way, TAP Streams generated using Perl done_testing() now works well with the plug-in
Version 1.2.6
- Support JSON within YAMLish data
Version 1.1
- Support to Bail out!'s
- JENKINS-10562 TAP Plugin fails on slave
Version 1.0
- Initial design of the plug-in
- Custom UI for TAP test results
- JUnit-like graph that displays the test results per build (actually the graph was modeled using TestNG Plug-in as basis)
Roadmap (wish list)
Version 1.x
- Add configurations like validate number of tests with test plan, a TODO causes a test to fail
- Diagnostics image gallery Done! Fixed as a new plug-in
- Diagnostics exception code formatting
- Add a link to open the file in the Build workspace (think about remote and local issues) Done!
History
The idea of the plug-in surged after tap4j was created. After learning about Smolder, it became evident that Jenkins could be used as a replacement for it. All that was needed was just adding TAP support to Jenkins and implementing a nice UI to display the test results. After some messages in jenkins-dev-list, Max and Nick commented about their need to show test results in a different manner than how Jenkins was doing at that moment. Soon after that Max, Nick, Bruno (tap4j) and Cesar (tap4j) started to work together, exchanging mail messages and discussing about a initial design for this plug-in.
In July 2011 the first version of the plug-in was ready to be released. The graph code used here was adapted from TestNG Plugin (big thanks to the development team, great work). The diagnostic (YAMLish) was implemented in Jelly + Java + CSS. And the road map was incremented based on what Gábor Szabó posted about Smolder and testing reports in his blog (see resources for links).
Simulating TAP streams with shell build steps
#!/bin/bash for x in {1..100}; do cat > $x.tap <<EOF ok 1 not ok 2 ok 3 # SKIP not ok 4 ok 5 ok 6 ok 7 EOF done ;