Description

This plugin collects and publishes Robot Framework test results in Jenkins.
Preconditions: all "suggested plugins" are "installed" in Jenkins.

Issue Tracker

Current features

• Test suite and case details with trend graphs
• Environment variable expansion for build paths in configuration
• Wildcard support for output files for parsing and archiving of multiple Robot result files
• Collection and archiving of original Robot report HTML files
• Summary of Robot run on build page
• Summary of latest Robot run on project page
• Trend graph of passed tests over builds on project page
• Marking build failed/unstable/successful according to pass thresholds given by user
• Option of evaluating only critical tests against the pass thresholds
• Listview column to show overall passed/failed tests in project listing and duration trend
• Configurable archiving of arbitrary Robot related artifacts per build 
• Token macros for usage with e.g. email-ext plugin
• Test visibility in radiator views

Configuration

Basic configuration of testing task

1. Configure your project
2. Select: Build -> Add build step -> execute shell/Execute Windows batch command
3. Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
4. Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)
5. Select: Post-build Actions -> Publish Robot Framework test results
6. Set path where your results are located (in above example command "my_robot_results")
7. If your output files are named differently than default (i.e. output.xml, report.html and log.html) specify the filenames by pressing the "Advanced..." button and writing the names in relevant fields. The fields support wildcards * and ?.
8. If you have other artifacts such as screenshots that you want to persist for viewing later in the logs you must configure them under "Advanced... -> Other files to copy". The field accepts comma separated list of files and supports wildcards * and ?.
9. Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.

Configuring direct links to log files

Version 1.3.0

Links are automatically generated to whatever files are configured to "Report html" and "Log html" in the plugin configuration. Links to log and report htmls are now only in the summary sections in the middle of the page and the sidebar link from previous versions have been removed.

Version 1.2.3 and earlier

From version 1.2 it's possible to configure a direct link from both sidepanel and project/build page summaries to chosen file persisted with the build. If no file is configured the link won't show up. Steps to do this are as follows:

1. Go to project configuration
2. Configure the name of the file to be linked to Log/Report link field
3. Make sure that you are saving the actual file you want to link with the build by checking the filenames/masks configured in the fields behind "Advanced..." button. By default these are output.xml, report.html and log.html which are the Robot Framework default names
4. Save the configuration and run a build. The links are persisted per build in order to withstand changes in file naming. Thus the links won't appear in builds that exist already.
5. Links should appear in the project/build summaries and in build sidepanel.

NOTE: After configuration change etc. the link will appear also in project sidepanel, but for now it doesn't appear there right after the build. Conversely, the project view sidepanel link won't go away until after configuration change/resave in project. We're working on a solution to this.

Sidepanel link

Summary link

Configuring Robot overall pass/fail to show in the project list

1. Go to Jenkins front page
2. Select the + sign in the top tabs of the project listing to create a new view or go to http://YOURJENKINSHOST/newView
3. Name your view and proceed creating a list view
4. Choose project you want to include to the view. By default the "Robot pass/fail" column will appear to the column listing.
5. Save view and select it from the top tabs in Jenkins project listing
6. (Optional) If you want to have the view on by default go to "Manage Jenkins"->"Configure System" (or http://YOURJENKINSHOST/configure )and select your newly created view to be the default one.

List view column in action

Using token macros in e-mail report

Prerequisites: token-macro plugin and email-ext plugin installed.

1. Go to project configuration
2. Enable "Editable Email Notification" in post-build actions (configuration reference)
3. Now you can use the following Robot variables in your custom result email:

• ${ROBOT_FAILEDCASES} - Expands to list of failed Robot cases. Each case on its own line.
• ${ROBOT_PASSPERCENTAGE, onlyCritical} - Expands to pass percentage of tests. (passed / total * 100 %),
  onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
• ${ROBOT_PASSRATIO, onlyCritical} Expands to build result in 'passed / total' format.
  onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
• ${ROBOT_REPORTLINK} - If logfile link is configured in the Robot plugin this link will point to that file for the build. Else show link to Robot reports directory for the build.

Displaying test numbers in build radiator views etc.

Overall Screenshots

Config

Project view

Build view

Detailed build view

Bugs

List of open bugs on robot-plugin

Changelog

Release 2.0.0

• Pull request 18 New design for list view results column
• Pull request 22 Support for `rebot` merged output.xml file
• Pull request 23 Add pipeline step support
• Pull request 24 Fix empty result in REST API
• Pull request 25 Add suite and test descriptions to results view

Release 1.6.5

• Pull request 15 Fix for evaluating case as critically failed
• Pull request 16 Removes second reading of entire history in createDurationGraphForTestObject
• Pull request 19 Changed new Robot Icons
• Pull request 20 [FIXED JENKINS-47227] Add public getEnableCache
• Pull request 21 Add Jenkinsfile
• JENKINS-52447 Incorrect reporting of total test cases

Release 1.6.4

• Pull request 12 Support Jenkins Workflow

• Pull request 13 Make cache based on WeakReferences optional

• Pull request 14 handle empty or invalid maxBuildToShow parameter

Release 1.6.2

• JENKINS-29353 Configurable number of builds for trends generation

Release 1.6.1

• Reverted earlier caching changes, because sometimes it increased memory usage too much
• JENKINS-29178 Robot Framework 2.9 output parsing is broken

Release 1.6.0

• Performance enhancements by improved result caching
• JENKINS-27141 Option to show only critical tests in graphs
• JENKINS-26980 Enable configuration of "Robot Results" column in the default job list view
• JENKINS-24868 Add the ability for robot plugin graph to show only failed tests

Release 1.5.0

• JENKINS-24916 Link to robot log from individual tests and suites
• JENKINS-24792 Cleanup graph zooming
• JENKINS-24688 Robot framework summary split in multiple rows in job list

Release 1.4.3

• JENKINS-24531 There should be high resolution versions of graphs available
• JENKINS-22406 Show tags for test cases
• JENKINS-21739 Links to robot framework report files are invalid when inside a folder
• JENKINS-21644 Aggregated results in a multimodule maven project displays only values of the last submodule

Release 1.4.2

• JENKINS-22060 Plugin does not count suite setup times into duration
• JENKINS-21736 Make the archiving of output.xml optional

Release 1.4.1

• JENKINS-21489 URL decoding for test objects fails with special characters
• JENKINS-21490 Split token-macro tokens further to enable more custom reporting

Release 1.4.0

• JENKINS-8381 Robot summary and trend graph for multi-configuration projects

Release 1.3.2

• JENKINS-19479 Robot Framework Plugin Threshold uses rounded percentage for evaluating pass/unstable build status
• JENKINS-19484 Robot Framework plugin shows result as "Failed!" on individual test case result page

Release 1.3.1

• JENKINS-19328 Whole directory is copied when "Other files to copy" -field is empty

Release 1.3.0

• JENKINS-15809 Failed to scout hudson.plugins.robot.tokens.RobotPassPercentageTokenMacro
• JENKINS-15188 Human readable duration values for test summaries and trend graphs
• JENKINS-15191 Robot icon is scaled ugly in many places
• JENKINS-15193 Graph scaling to see the significant changes in test count / duration
• JENKINS-12795 Log/Report link in Job page is broken when Jenkins is started with a specific prefix
• JENKINS-19064 Project/build page summary improvements
• JENKINS-15768 Plug-in doesn't retain directories when copy files using "Other files to copy" setting
• JENKINS-16615 Link to log.html is broken on project page in matrix project
• JENKINS-18675 Failure in suite teardown does not fail test cases
• JENKINS-17328 output.xml kept open after build if parsing error occurs causing subsequent runs to fail
• JENKINS-15327 Publish Robot Framework test results not support chinese character

Release 1.2.3

• JENKINS-15194 - Basic token macros created for use in e.g. email-ext plugin
• JENKINS-15187 - Made the plugin report tests as unit tests in order to see test count in different views e.g. radiator plugins (see config notice above)

Release 1.2.2

• Changed XML parsing to much more memory efficient StAX. Should solve many outOfMemoryErrors.
• Changed the way suites with same names are handled
• Fixed a bug where empty "Other files to copy" field leads to copying the whole directory into saved builds.

Release 1.2.1

• Compliance with RF 2.6 log file changes (html and javascript files separated)
• Option to save arbitrary test artifacts with test results

Release 1.2

• configurable file link (to log / report file) on project and build pages
• Listviewcolumn to show overall passed/failed tests in project listing
• pass/fail graph to testcase view
• support for changed robot output xml (issue: http://www.google.com/url?sa=D&q=http://code.google.com/p/robotframework/issues/detail%3Fid%3D820)

Release 1.1

• JENKINS-9078 - Jenkins environment variables available for configurations
• JENKINS-9079 - Support for GLOB style wildcards in file configurations
• Detailed views of test cases and suites with trend graphs
• Fix to JDK 1.5 incompliant exception throw

Release 1.0

• Initial release of the plugin