Jenkins : Robot Framework Plugin

Created by Unknown User (janne.piironen@gmail.com), last modified by Unknown User (tattoo) on Sep 27, 2019

Plugin Information

View Robot Framework on the plugin site for more information.

Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:


Description

This plugin collects and publishes Robot Framework test results in Jenkins.
Preconditions: all "suggested plugins" are "installed" in Jenkins.

Issue Tracker

Current features

  • Test suite and case details with trend graphs
  • Environment variable expansion for build paths in configuration
  • Wildcard support for output files for parsing and archiving of multiple Robot result files
  • Collection and archiving of original Robot report HTML files
  • Summary of Robot run on build page
  • Summary of latest Robot run on project page
  • Trend graph of passed tests over builds on project page
  • Marking build failed/unstable/successful according to pass thresholds given by user
  • Option of evaluating only critical tests against the pass thresholds
  • Listview column to show overall passed/failed tests in project listing and duration trend
  • Configurable archiving of arbitrary Robot related artifacts per build 
  • Token macros for usage with e.g. email-ext plugin
  • Test visibility in radiator views

Configuration

Basic configuration of testing task

  1. Configure your project
  2. Select: Build -> Add build step -> execute shell/Execute Windows batch command
  3. Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
  4. Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)
  5. Select: Post-build Actions -> Publish Robot Framework test results
  6. Set path where your results are located (in above example command "my_robot_results")
  7. If your output files are named differently than default (i.e. output.xml, report.html and log.html) specify the filenames by pressing the "Advanced..." button and writing the names in relevant fields. The fields support wildcards * and ?.
  8. If you have other artifacts such as screenshots that you want to persist for viewing later in the logs you must configure them under "Advanced... -> Other files to copy". The field accepts comma separated list of files and supports wildcards * and ?.
  9. Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.

Configuring direct links to log files

Version 1.3.0

Links are automatically generated to whatever files are configured to "Report html" and "Log html" in the plugin configuration. Links to log and report htmls are now only in the summary sections in the middle of the page and the sidebar link from previous versions have been removed.

Version 1.2.3 and earlier

From version 1.2 it's possible to configure a direct link from both sidepanel and project/build page summaries to chosen file persisted with the build. If no file is configured the link won't show up. Steps to do this are as follows:

  1. Go to project configuration
  2. Configure the name of the file to be linked to Log/Report link field
  3. Make sure that you are saving the actual file you want to link with the build by checking the filenames/masks configured in the fields behind "Advanced..." button. By default these are output.xml, report.html and log.html which are the Robot Framework default names
  4. Save the configuration and run a build. The links are persisted per build in order to withstand changes in file naming. Thus the links won't appear in builds that exist already.
  5. Links should appear in the project/build summaries and in build sidepanel.

NOTE: After configuration change etc. the link will appear also in project sidepanel, but for now it doesn't appear there right after the build. Conversely, the project view sidepanel link won't go away until after configuration change/resave in project. We're working on a solution to this.


Sidepanel link


Summary link

Configuring Robot overall pass/fail to show in the project list

  1. Go to Jenkins front page
  2. Select the + sign in the top tabs of the project listing to create a new view or go to http://YOURJENKINSHOST/newView
  3. Name your view and proceed creating a list view
  4. Choose project you want to include to the view. By default the "Robot pass/fail" column will appear to the column listing.
  5. Save view and select it from the top tabs in Jenkins project listing
  6. (Optional) If you want to have the view on by default go to "Manage Jenkins"->"Configure System" (or http://YOURJENKINSHOST/configure )and select your newly created view to be the default one.


List view column in action

Using token macros in e-mail report

Prerequisites: token-macro plugin and email-ext plugin installed.

  1. Go to project configuration
  2. Enable "Editable Email Notification" in post-build actions (configuration reference)
  3. Now you can use the following Robot variables in your custom result email:
  • ${ROBOT_FAILEDCASES} - Expands to list of failed Robot cases. Each case on its own line.
  • ${ROBOT_PASSPERCENTAGE, onlyCritical} - Expands to pass percentage of tests. (passed / total * 100 %),
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_PASSRATIO, onlyCritical} Expands to build result in 'passed / total' format.
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_REPORTLINK} - If logfile link is configured in the Robot plugin this link will point to that file for the build. Else show link to Robot reports directory for the build.

Displaying test numbers in build radiator views etc.

If the Robot plugin marks the build as failure the tests will not show up. This is because only unstable and successful builds are considered to have test results per Jenkins.

This means that in order to see the test results in other views you must set your unstable threshold so that the build never goes to failure.

Overall Screenshots

Config

Project view

Build view

Detailed build view

Bugs

List of open bugs on robot-plugin

Key Summary T Created Updated P Status
JENKINS-37739 Support multiple output.xml's to support parallel test executions in pipeline Improvement Aug 28, 2016 Sep 03, 2021 Trivial Open
JENKINS-66150 Issue on output.xml parser Bug Jul 15, 2021 Aug 03, 2021 Critical Open
JENKINS-66032 ${ROBOT_PASSPERCENTAGE} for test suite New Feature Jul 02, 2021 Aug 02, 2021 Critical Open
JENKINS-63900 Robot Framework results trend graph is not showing the correct build number of the job Bug Oct 09, 2020 Jun 21, 2021 Minor Reopened
JENKINS-23077 New configuration option: do not throw exception about missing output.xml when the build fails Improvement May 16, 2014 May 04, 2021 Major Open
JENKINS-26245 Add option to show "all tests" or "critical tests" in the Listviewcolumn that shows overall passed/failed tests in project listing Bug Dec 30, 2014 May 04, 2021 Minor Reopened
JENKINS-41560 Robot Framework plugin on Windows not honoring passed/unstable percentages Bug Jan 30, 2017 Apr 27, 2021 Major Open
JENKINS-15190 Creation of trend graphs is slow on large number of builds Improvement Sep 17, 2012 Apr 21, 2021 Minor Open
JENKINS-11441 Aggregate Downstream Robot Reports Improvement Oct 22, 2011 Mar 02, 2021 Minor Open
JENKINS-8378 Filter test chart New Feature Dec 29, 2010 Mar 02, 2021 Minor Open
JENKINS-8380 Allow clean shutdown of testing from UI New Feature Dec 30, 2010 Mar 02, 2021 Minor Open
JENKINS-15189 Make Robot plugin influence the weather icon New Feature Sep 17, 2012 Mar 02, 2021 Minor Open
JENKINS-15195 Send old 'Summary Report' style report to e-mail Improvement Sep 17, 2012 Mar 02, 2021 Minor Open
JENKINS-15200 Test case history in matrix format New Feature Sep 17, 2012 Mar 02, 2021 Minor Open
JENKINS-18840 Remove support for Robot Framework split outputs. Improvement Jul 19, 2013 Mar 02, 2021 Minor Open
JENKINS-22603 Add support Dashboard View New Feature Apr 14, 2014 Mar 02, 2021 Minor Open
JENKINS-33329 Robot-plugin incorrectly recognized CloudBees Folder project to display Robot results column Bug Mar 05, 2016 Feb 24, 2021 Major Open
JENKINS-56862 Provide support to claim failing test(s). New Feature Apr 03, 2019 Apr 03, 2019 Minor Open
JENKINS-55675 Add RF RPA support Improvement Jan 18, 2019 Jan 18, 2019 Major Open
JENKINS-24968 Show documentation on suite and test view Improvement Oct 02, 2014 Dec 01, 2017 Minor Open
Authenticate to retrieve your issues
20 issues

Changelog

Release 2.0.0

Release 1.6.5

Release 1.6.4

Release 1.6.2

Release 1.6.1

Release 1.6.0

Release 1.5.0

Release 1.4.3

Release 1.4.2

  • JENKINS-22060 Plugin does not count suite setup times into duration
  • JENKINS-21736 Make the archiving of output.xml optional

Release 1.4.1

  • JENKINS-21489 URL decoding for test objects fails with special characters
  • JENKINS-21490 Split token-macro tokens further to enable more custom reporting

Release 1.4.0

  • JENKINS-8381 Robot summary and trend graph for multi-configuration projects

Release 1.3.2

  • JENKINS-19479 Robot Framework Plugin Threshold uses rounded percentage for evaluating pass/unstable build status
  • JENKINS-19484 Robot Framework plugin shows result as "Failed!" on individual test case result page

Release 1.3.1

  • JENKINS-19328 Whole directory is copied when "Other files to copy" -field is empty

Release 1.3.0

  • JENKINS-15809 Failed to scout hudson.plugins.robot.tokens.RobotPassPercentageTokenMacro
  • JENKINS-15188 Human readable duration values for test summaries and trend graphs
  • JENKINS-15191 Robot icon is scaled ugly in many places
  • JENKINS-15193 Graph scaling to see the significant changes in test count / duration
  • JENKINS-12795 Log/Report link in Job page is broken when Jenkins is started with a specific prefix
  • JENKINS-19064 Project/build page summary improvements
  • JENKINS-15768 Plug-in doesn't retain directories when copy files using "Other files to copy" setting
  • JENKINS-16615 Link to log.html is broken on project page in matrix project
  • JENKINS-18675 Failure in suite teardown does not fail test cases
  • JENKINS-17328 output.xml kept open after build if parsing error occurs causing subsequent runs to fail
  • JENKINS-15327 Publish Robot Framework test results not support chinese character

Release 1.2.3

  • JENKINS-15194 - Basic token macros created for use in e.g. email-ext plugin
  • JENKINS-15187 - Made the plugin report tests as unit tests in order to see test count in different views e.g. radiator plugins (see config notice above)

Release 1.2.2

  • Changed XML parsing to much more memory efficient StAX. Should solve many outOfMemoryErrors.
  • Changed the way suites with same names are handled
  • Fixed a bug where empty "Other files to copy" field leads to copying the whole directory into saved builds.

Release 1.2.1

  • Compliance with RF 2.6 log file changes (html and javascript files separated)
  • Option to save arbitrary test artifacts with test results

Release 1.2

Release 1.1

  • JENKINS-9078 - Jenkins environment variables available for configurations
  • JENKINS-9079 - Support for GLOB style wildcards in file configurations
  • Detailed views of test cases and suites with trend graphs
  • Fix to JDK 1.5 incompliant exception throw

Release 1.0

  • Initial release of the plugin

Attachments:

config.png (image/png)
config.png (image/png)
config.png (image/png)
config1.png (image/png)
project_page.png (image/png)
build_page.png (image/png)
detailed.png (image/png)
robo_loglink_projectpage.png (image/png)
loghtml_sidepanel.png (image/png)
logfile_configuration_screen.png (image/png)
robot_view_column.png (image/png)
ride.jpg (image/jpeg)
jenkins.jpg (image/jpeg)
how_to_find_robot_error_log.jpg (image/jpeg)
project_page.png (image/png)
build_page.png (image/png)
detailed.png (image/png)