Jenkins
Space shortcuts
Child pages
  • Robot Framework Plugin
Skip to end of metadata
Go to start of metadata

Plugin Information

View Robot Framework on the plugin site for more information.

 

Description

This plugin collects and publishes Robot Framework test results in Jenkins.
Preconditions: all "suggested plugins" are "installed" in Jenkins.

Issue Tracker

Current features

  • Test suite and case details with trend graphs
  • Environment variable expansion for build paths in configuration
  • Wildcard support for output files for parsing and archiving of multiple Robot result files
  • Collection and archiving of original Robot report HTML files
  • Summary of Robot run on build page
  • Summary of latest Robot run on project page
  • Trend graph of passed tests over builds on project page
  • Marking build failed/unstable/successful according to pass thresholds given by user
  • Option of evaluating only critical tests against the pass thresholds
  • Listviewcolumn to show overall passed/failed tests in project listing
  • Configurable archiving of arbitrary Robot related artifacts per build 
  • Token macros for usage with e.g. email-ext plugin
  • Test visibility in radiator views

Configuration

Basic configuration of testing task

  1. Configure your project
  2. Select: Build -> Add build step -> execute shell/Execute Windows batch command
  3. Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
  4. Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)
  5. Select: Post-build Actions -> Publish Robot Framework test results
  6. Set path where your results are located (in above example command "my_robot_results")
  7. If your output files are named differently than default (i.e. output.xml, report.html and log.html) specify the filenames by pressing the "Advanced..." button and writing the names in relevant fields. The fields support wildcards * and ?.
  8. If you have other artifacts such as screenshots that you want to persist for viewing later in the logs you must configure them under "Advanced... -> Other files to copy". The field accepts comma separated list of files and supports wildcards * and ?.
  9. Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.

Configuring direct links to log files

Version 1.3.0

Links are automatically generated to whatever files are configured to "Report html" and "Log html" in the plugin configuration. Links to log and report htmls are now only in the summary sections in the middle of the page and the sidebar link from previous versions have been removed.

Version 1.2.3 and earlier

From version 1.2 it's possible to configure a direct link from both sidepanel and project/build page summaries to chosen file persisted with the build. If no file is configured the link won't show up. Steps to do this are as follows:

  1. Go to project configuration
  2. Configure the name of the file to be linked to Log/Report link field
  3. Make sure that you are saving the actual file you want to link with the build by checking the filenames/masks configured in the fields behind "Advanced..." button. By default these are output.xml, report.html and log.html which are the Robot Framework default names
  4. Save the configuration and run a build. The links are persisted per build in order to withstand changes in file naming. Thus the links won't appear in builds that exist already.
  5. Links should appear in the project/build summaries and in build sidepanel.

NOTE: After configuration change etc. the link will appear also in project sidepanel, but for now it doesn't appear there right after the build. Conversely, the project view sidepanel link won't go away until after configuration change/resave in project. We're working on a solution to this.


Sidepanel link


Summary link

Configuring Robot overall pass/fail to show in the project list

  1. Go to Jenkins front page
  2. Select the + sign in the top tabs of the project listing to create a new view or go to http://YOURJENKINSHOST/newView
  3. Name your view and proceed creating a list view
  4. Choose project you want to include to the view. By default the "Robot pass/fail" column will appear to the column listing.
  5. Save view and select it from the top tabs in Jenkins project listing
  6. (Optional) If you want to have the view on by default go to "Manage Jenkins"->"Configure System" (or http://YOURJENKINSHOST/configure )and select your newly created view to be the default one.


List view column in action

Using token macros in e-mail report

Prerequisites: token-macro plugin and email-ext plugin installed.

  1. Go to project configuration
  2. Enable "Editable Email Notification" in post-build actions (configuration reference)
  3. Now you can use the following Robot variables in your custom result email:
  • ${ROBOT_FAILEDCASES} - Expands to list of failed Robot cases. Each case on its own line.
  • ${ROBOT_PASSPERCENTAGE, onlyCritical} - Expands to pass percentage of tests. (passed / total * 100 %),
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_PASSRATIO, onlyCritical} Expands to build result in 'passed / total' format.
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_REPORTLINK} - If logfile link is configured in the Robot plugin this link will point to that file for the build. Else show link to Robot reports directory for the build.

Displaying test numbers in build radiator views etc.

If the Robot plugin marks the build as failure the tests will not show up. This is because only unstable and successful builds are considered to have test results per Jenkins.

This means that in order to see the test results in other views you must set your unstable threshold so that the build never goes to failure.

Overall Screenshots

Config

Project view

Build view

Detailed build view

Bugs

List of open bugs on robot-plugin

Key Summary T Created Updated P Status
Refresh

Changelog

Release 1.6.5

Release 1.6.4

Release 1.6.2

Release 1.6.1

Release 1.6.0

Release 1.5.0

Release 1.4.3

Release 1.4.2

  • JENKINS-22060 Plugin does not count suite setup times into duration
  • JENKINS-21736 Make the archiving of output.xml optional

Release 1.4.1

  • JENKINS-21489 URL decoding for test objects fails with special characters
  • JENKINS-21490 Split token-macro tokens further to enable more custom reporting

Release 1.4.0

  • JENKINS-8381 Robot summary and trend graph for multi-configuration projects

Release 1.3.2

  • JENKINS-19479 Robot Framework Plugin Threshold uses rounded percentage for evaluating pass/unstable build status
  • JENKINS-19484 Robot Framework plugin shows result as "Failed!" on individual test case result page

Release 1.3.1

  • JENKINS-19328 Whole directory is copied when "Other files to copy" -field is empty

Release 1.3.0

  • JENKINS-15809 Failed to scout hudson.plugins.robot.tokens.RobotPassPercentageTokenMacro
  • JENKINS-15188 Human readable duration values for test summaries and trend graphs
  • JENKINS-15191 Robot icon is scaled ugly in many places
  • JENKINS-15193 Graph scaling to see the significant changes in test count / duration
  • JENKINS-12795 Log/Report link in Job page is broken when Jenkins is started with a specific prefix
  • JENKINS-19064 Project/build page summary improvements
  • JENKINS-15768 Plug-in doesn't retain directories when copy files using "Other files to copy" setting
  • JENKINS-16615 Link to log.html is broken on project page in matrix project
  • JENKINS-18675 Failure in suite teardown does not fail test cases
  • JENKINS-17328 output.xml kept open after build if parsing error occurs causing subsequent runs to fail
  • JENKINS-15327 Publish Robot Framework test results not support chinese character

Release 1.2.3

  • JENKINS-15194 - Basic token macros created for use in e.g. email-ext plugin
  • JENKINS-15187 - Made the plugin report tests as unit tests in order to see test count in different views e.g. radiator plugins (see config notice above)

Release 1.2.2

  • Changed XML parsing to much more memory efficient StAX. Should solve many outOfMemoryErrors.
  • Changed the way suites with same names are handled
  • Fixed a bug where empty "Other files to copy" field leads to copying the whole directory into saved builds.

Release 1.2.1

  • Compliance with RF 2.6 log file changes (html and javascript files separated)
  • Option to save arbitrary test artifacts with test results

Release 1.2

Release 1.1

  • JENKINS-9078 - Jenkins environment variables available for configurations
  • JENKINS-9079 - Support for GLOB style wildcards in file configurations
  • Detailed views of test cases and suites with trend graphs
  • Fix to JDK 1.5 incompliant exception throw

Release 1.0

  • Initial release of the plugin

28 Comments

  1. jrvilda -

    Link do not work

    Issue Tracking

    Open Issues (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+'robot')

    . Link to "robot", rather than "robot-plugin".

    Issue Tracking

    Open Issues (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+'robot-plugin')

    Changing the ID, not shown the table.

  2. Eric Stob

    This plugin is awesome, thank you!

  3. Dave james

    Is it possible to use this plugin for Rational Functional Tester?

  4. Charlie Chen

    Open Latest Robot report.html link in project page is not linked to the latest Robot report. User has to go into indivdual build and click Open Robot report.html link in there.

  5. Miltos kollin

    The links is not work guys. robot plugin

  6. Greg Pugh

    Why would I get this error?  The script runs fine.

    Started by user xxx

    Building on master
    workspace $ cmd /c call "e:\Program Files\Apache Software Foundation\Tomcat6\temp\hudson4620332721957956251.bat"

    e:\.jenkins\jobs\test\workspace>start call pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt

    e:\.jenkins\jobs\test\workspace>exit 0
    Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information
    Robot results publisher started...
    -Parsing output xml:
    Failed!
    java.io.IOException: Error on line 195 of document file:///E:/Reports/Secretariat/output_xxx-20110818-142217.xml : XML document structures must start and end within the same entity. Nested exception: XML document structures must start and end within the same entity.
    at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:71)
    at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:44)
    at hudson.FilePath.act(FilePath.java:758)
    at hudson.FilePath.act(FilePath.java:740)
    at hudson.plugins.robot.RobotParser.parse(RobotParser.java:39)
    at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:235)
    at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:258)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
    at hudson.model.Build$RunnerImpl.post2(Build.java:161)
    at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
    at hudson.model.Run.run(Run.java:1400)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:175)
    Build step 'Publish Robot Framework test results' changed build result to FAILURE
    Finished: FAILURE

    ISSUE SOLVED

    Changed batch command from

    start call pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt

    to

    pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt

  7. Laurent Bristiel

    Hello,

    I have a problem with error message display within the plugin.

    My situation is : 
    - I use of the last version of the softwares (Robot, Ride, plug-in, Python)
    - When a Robot test case fails I raise an error with a multi-line messages (to see different errors)
    those multi-lines strings are created using '\r\n' for cariage return (I tried '\n' before but result was the same)

    My problem :
    - Error message does display in multi-line in Ride or in the Robot output, but when I look at it within the Jenkins plugin, it is all in a single line.

    See screenshot bellow.

    Does anyopne has an idea of the problem ?

    Thanks,
    Laurent

    via jenkins plugin : (not nice, not carriage return)
    via RIDE/ROBOT (nice : carriage return handled)

  8. Jose Sa

    Is there any way to Aggregate Donwstream Robot Reports the same way it does for JUnit reports?

    My idea is to have in the compilation job where the "installable units" are generated which will trigger the sucession of robot executions (bringup -> smoke -> regression, ..) to have aggregated the robot results of all downstream jobs in order to evaluate the quality of the artifacts in a single place.

  9. ben li

    Hi,

    by robot-plugin, we can get test result overview from "Robot Framework test results" per build, and it can show all failed case in "All failed testcases". And my question is that I can not get the log file of each failed case in "All failed testcases"? is there any links? how can I configure it? I can only get a short message like followings.

  10. Michael Driscoll

    The plugin works great for us, would it be possible to add a class to the "Robot Results" column td so that it can be styled differently?

  11. Jose Sa

    Is there any REST api planned for this plugin ?

  12. Tiago Oliveira

    The plugin is failing if output.xml was created from several merged output.xml (using rebot --merge)

    1. David Barba

      I have the same problem. And give me the following failure:16531 Killed rebot -L DEBUG --nostatusrc --removekeywords wuks --removekeywords passed --report $WORKSPACE/report_merge.html --output $WORKSPACE/output_merge.xml --log $WORKSPACE/log_merge.html --xunit $WORKSPACE/xunitResults_merge.xml --merge $WORKSPACE/output.xml $WORKSPACE/output2nd.xml
      Error merging

      Do you found any solution?

  13. yao wang

    Is there any Environment variables for the result? Such as BUILD_NUMBER means the Build number that I can use this variable in Email notification plugin.

    Now, I want to use Email-ext plugin to send Robot framework result in email content, like:

    The automation result is XXX total | XX pass | XX failed.

    The failed cases are XXX XXX XXX. 

  14. Ed Nonog

    Encountered this error in opening the report of Robot Framework. Thanks in advance. :D

    Opening Robot Framework report failed

    • Verify that you have JavaScript enabled in your browser.
    • Make sure you are using a modern enough browser. Firefox 3.5, IE 8, or equivalent is required, newer browsers are recommended.
    • Check are there messages in your browser's JavaScript error log. Please report the problem if you suspect you have encountered a bug.

  15. Michel Lawaty

    Hello,

    we use jenkins pipelines, and would like to integrate this plugin. We use it already for the normal jenkins job, but I wonder if someone is working on adapting to jenkinsfile/pipeline files.

    Any comments?

    cheers

    Michel

    1. Dave Bubenik

      Looks like you can, but there was something broken with the snippet generator. In the snippet generator it's under "step: General Build Step".

      I was able to publish robot results with the following snippet:

      step([$class: 'RobotPublisher', 
              outputPath: '', 
              outputFileName: 'output.xml', 
              reportFileName: 'report.html',
              logFileName: 'log.html',
              otherFiles: '',
              disableArchiveOutput: false,
              enableCache: true,
              unstableThreshold: 90,
              passThreshold: 95,
              onlyCritical: true
              ])

  16. Ari Mämmelä

    Hi,

     

    I've been trying to make automated Robot email reports through Jenkins and for that purpose I have used a Groovy-script. So far I've been successful on making pretty good looking reports in HTML format but haven't yet figured how to get the "warning/error" information attached to the failed test cases.

    Has anyone else been tackling with this problem?

  17. li zhe

    Hello

    My Environment is below, The jenkins was installed on ubuntu, My PC(Windows7) as slave in jenkins. I using agnet by Java Web to via windows.

    I was run windows batch command "pybot.bat --version" in slave. I got below issue:


    E:\Jenkins\workspace\RF_Windows>pybot.bat --version
    Robot Framework 3.0.2 (Python 2.7.14 on win32)
    Build step 'Execute Windows batch command' marked build as failure
    Finished: FAILURE

    1. Tatu Kairi

      The plugin expects you to have output.xml, log.html and report.html files existing in post-build step. {{pybot.bat --version}} naturally does not produce the files.

  18. George Quintas

    Hi,

    I installed this plugin with Jenkins on Linux.


    Your step below do not work:

    3.) Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.


    it's saying "not found".


    Regards,

    George

    1. Tatu Kairi

      Hi George Quintas,


      Please follow Robot Framework's instructions how to install and how to run the actual tool. This plugin operates on the various result files Robot Framework produces.  


      BR,

      -TKa

  19. arthur chang

    Hi,

    I installed Jenkins 2.150.3 and robot framework plugin 1.6.5, but there is no post build action "publish robot framework tests result". What should I do if I want to use robot plugin for Jenkins 2? 


    Best Regards,

    Arthur

    1. Tatu Kairi

      Hi arthur chang ,


      Something has gone wrong with the installation of the plugin. Please re-install the plugin and restart Jenkins.


      BR,

      -TKa

      1. arthur chang

        Hi Tatu Kairi,


        Thank you for your reply, I reinstall the plugin and restart Jenkins, it works now.


        Thanks & Regards,

        Arthur

  20. Alokkumar Mahajan

    Hello,

    I have request to claim failed tests from Robot Framework tests reports in addition to claiming the failed job. i see this feature is available for Junit Plugin JUnit Plugin . Can we have this feature added as part of Robot Framework plugin as well?

    Thanks,

    -alok

    1. Tatu Kairi

      Hi Alokkumar Mahajan,


      Please submit bugs or feature requests to the issue tracker. They won't otherwise be considered at all. 

      1. Alokkumar Mahajan

        Tatu Kairi Thanks, i tried searching if there's any open request for this but couldn't find anything related. So, created Feature request  JENKINS-56862 - Getting issue details... STATUS  


        Regards,

        ~alok