Plugin Information |
---|
View Robot Framework on the plugin site for more information. |
Description
This plugin collects and publishes Robot Framework test results in Jenkins.
Preconditions: all "suggested plugins" are "installed" in Jenkins.
Current features
- Test suite and case details with trend graphs
- Environment variable expansion for build paths in configuration
- Wildcard support for output files for parsing and archiving of multiple Robot result files
- Collection and archiving of original Robot report HTML files
- Summary of Robot run on build page
- Summary of latest Robot run on project page
- Trend graph of passed tests over builds on project page
- Marking build failed/unstable/successful according to pass thresholds given by user
- Option of evaluating only critical tests against the pass thresholds
- Listview column to show overall passed/failed tests in project listing and duration trend
- Configurable archiving of arbitrary Robot related artifacts per build
- Token macros for usage with e.g. email-ext plugin
- Test visibility in radiator views
Configuration
Basic configuration of testing task
- Configure your project
- Select: Build -> Add build step -> execute shell/Execute Windows batch command
- Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
- Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)
- Select: Post-build Actions -> Publish Robot Framework test results
- Set path where your results are located (in above example command "my_robot_results")
- If your output files are named differently than default (i.e. output.xml, report.html and log.html) specify the filenames by pressing the "Advanced..." button and writing the names in relevant fields. The fields support wildcards * and ?.
- If you have other artifacts such as screenshots that you want to persist for viewing later in the logs you must configure them under "Advanced... -> Other files to copy". The field accepts comma separated list of files and supports wildcards * and ?.
- Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.
Configuring direct links to log files
Version 1.3.0
Links are automatically generated to whatever files are configured to "Report html" and "Log html" in the plugin configuration. Links to log and report htmls are now only in the summary sections in the middle of the page and the sidebar link from previous versions have been removed.
Version 1.2.3 and earlier
From version 1.2 it's possible to configure a direct link from both sidepanel and project/build page summaries to chosen file persisted with the build. If no file is configured the link won't show up. Steps to do this are as follows:
- Go to project configuration
- Configure the name of the file to be linked to Log/Report link field
- Make sure that you are saving the actual file you want to link with the build by checking the filenames/masks configured in the fields behind "Advanced..." button. By default these are output.xml, report.html and log.html which are the Robot Framework default names
- Save the configuration and run a build. The links are persisted per build in order to withstand changes in file naming. Thus the links won't appear in builds that exist already.
- Links should appear in the project/build summaries and in build sidepanel.
NOTE: After configuration change etc. the link will appear also in project sidepanel, but for now it doesn't appear there right after the build. Conversely, the project view sidepanel link won't go away until after configuration change/resave in project. We're working on a solution to this.
Sidepanel link
Summary link
Configuring Robot overall pass/fail to show in the project list
- Go to Jenkins front page
- Select the + sign in the top tabs of the project listing to create a new view or go to http://YOURJENKINSHOST/newView
- Name your view and proceed creating a list view
- Choose project you want to include to the view. By default the "Robot pass/fail" column will appear to the column listing.
- Save view and select it from the top tabs in Jenkins project listing
- (Optional) If you want to have the view on by default go to "Manage Jenkins"->"Configure System" (or http://YOURJENKINSHOST/configure )and select your newly created view to be the default one.
List view column in action
Using token macros in e-mail report
Prerequisites: token-macro plugin and email-ext plugin installed.
- Go to project configuration
- Enable "Editable Email Notification" in post-build actions (configuration reference)
- Now you can use the following Robot variables in your custom result email:
- ${ROBOT_FAILEDCASES} - Expands to list of failed Robot cases. Each case on its own line.
- ${ROBOT_PASSPERCENTAGE, onlyCritical} - Expands to pass percentage of tests. (passed / total * 100 %),
onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false. - ${ROBOT_PASSRATIO, onlyCritical} Expands to build result in 'passed / total' format.
onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false. - ${ROBOT_REPORTLINK} - If logfile link is configured in the Robot plugin this link will point to that file for the build. Else show link to Robot reports directory for the build.
Displaying test numbers in build radiator views etc.
If the Robot plugin marks the build as failure the tests will not show up. This is because only unstable and successful builds are considered to have test results per Jenkins.
This means that in order to see the test results in other views you must set your unstable threshold so that the build never goes to failure.
Overall Screenshots
Config
Project view
Build view
Detailed build view
Bugs
List of open bugs on robot-plugin
Changelog
Release 2.0.0
- Pull request 18 New design for list view results column
- Pull request 22 Support for `rebot` merged output.xml file
- Pull request 23 Add pipeline step support
- Pull request 24 Fix empty result in REST API
- Pull request 25 Add suite and test descriptions to results view
Release 1.6.5
- Pull request 15 Fix for evaluating case as critically failed
- Pull request 16 Removes second reading of entire history in createDurationGraphForTestObject
- Pull request 19 Changed new Robot Icons
- Pull request 20 [FIXED JENKINS-47227] Add public getEnableCache
- Pull request 21 Add Jenkinsfile
- JENKINS-52447 Incorrect reporting of total test cases
Release 1.6.4
- Pull request 12 Support Jenkins Workflow
- Pull request 13 Make cache based on WeakReferences optional
- Pull request 14 handle empty or invalid maxBuildToShow parameter
Release 1.6.2
- JENKINS-29353 Configurable number of builds for trends generation
Release 1.6.1
- Reverted earlier caching changes, because sometimes it increased memory usage too much
- JENKINS-29178 Robot Framework 2.9 output parsing is broken
Release 1.6.0
- Performance enhancements by improved result caching
- JENKINS-27141 Option to show only critical tests in graphs
- JENKINS-26980 Enable configuration of "Robot Results" column in the default job list view
- JENKINS-24868 Add the ability for robot plugin graph to show only failed tests
Release 1.5.0
- JENKINS-24916 Link to robot log from individual tests and suites
- JENKINS-24792 Cleanup graph zooming
- JENKINS-24688 Robot framework summary split in multiple rows in job list
Release 1.4.3
- JENKINS-24531 There should be high resolution versions of graphs available
- JENKINS-22406 Show tags for test cases
- JENKINS-21739 Links to robot framework report files are invalid when inside a folder
- JENKINS-21644 Aggregated results in a multimodule maven project displays only values of the last submodule
Release 1.4.2
- JENKINS-22060 Plugin does not count suite setup times into duration
- JENKINS-21736 Make the archiving of output.xml optional
Release 1.4.1
- JENKINS-21489 URL decoding for test objects fails with special characters
- JENKINS-21490 Split token-macro tokens further to enable more custom reporting
Release 1.4.0
- JENKINS-8381 Robot summary and trend graph for multi-configuration projects
Release 1.3.2
- JENKINS-19479 Robot Framework Plugin Threshold uses rounded percentage for evaluating pass/unstable build status
- JENKINS-19484 Robot Framework plugin shows result as "Failed!" on individual test case result page
Release 1.3.1
- JENKINS-19328 Whole directory is copied when "Other files to copy" -field is empty
Release 1.3.0
- JENKINS-15809 Failed to scout hudson.plugins.robot.tokens.RobotPassPercentageTokenMacro
- JENKINS-15188 Human readable duration values for test summaries and trend graphs
- JENKINS-15191 Robot icon is scaled ugly in many places
- JENKINS-15193 Graph scaling to see the significant changes in test count / duration
- JENKINS-12795 Log/Report link in Job page is broken when Jenkins is started with a specific prefix
- JENKINS-19064 Project/build page summary improvements
- JENKINS-15768 Plug-in doesn't retain directories when copy files using "Other files to copy" setting
- JENKINS-16615 Link to log.html is broken on project page in matrix project
- JENKINS-18675 Failure in suite teardown does not fail test cases
- JENKINS-17328 output.xml kept open after build if parsing error occurs causing subsequent runs to fail
- JENKINS-15327 Publish Robot Framework test results not support chinese character
Release 1.2.3
- JENKINS-15194 - Basic token macros created for use in e.g. email-ext plugin
- JENKINS-15187 - Made the plugin report tests as unit tests in order to see test count in different views e.g. radiator plugins (see config notice above)
Release 1.2.2
- Changed XML parsing to much more memory efficient StAX. Should solve many outOfMemoryErrors.
- Changed the way suites with same names are handled
- Fixed a bug where empty "Other files to copy" field leads to copying the whole directory into saved builds.
Release 1.2.1
- Compliance with RF 2.6 log file changes (html and javascript files separated)
- Option to save arbitrary test artifacts with test results
Release 1.2
- configurable file link (to log / report file) on project and build pages
- Listviewcolumn to show overall passed/failed tests in project listing
- pass/fail graph to testcase view
- support for changed robot output xml (issue: http://www.google.com/url?sa=D&q=http://code.google.com/p/robotframework/issues/detail%3Fid%3D820)
Release 1.1
- JENKINS-9078 - Jenkins environment variables available for configurations
- JENKINS-9079 - Support for GLOB style wildcards in file configurations
- Detailed views of test cases and suites with trend graphs
- Fix to JDK 1.5 incompliant exception throw
Release 1.0
- Initial release of the plugin
28 Comments
Unknown User (jrvilda)
Link do not work
Issue Tracking
Open Issues (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+'robot')
. Link to "robot", rather than "robot-plugin".
Issue Tracking
Open Issues (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+'robot-plugin')
Changing the ID, not shown the table.
Unknown User (ericstob)
This plugin is awesome, thank you!
Unknown User (davefeeder)
Is it possible to use this plugin for Rational Functional Tester?
Unknown User (isaac1944)
Open Latest Robot report.html link in project page is not linked to the latest Robot report. User has to go into indivdual build and click Open Robot report.html link in there.
Unknown User (miltos)
The links is not work guys. robot plugin
Unknown User (kohrock)
Why would I get this error? The script runs fine.
Started by user xxx
Building on master
workspace $ cmd /c call "e:\Program Files\Apache Software Foundation\Tomcat6\temp\hudson4620332721957956251.bat"
e:\.jenkins\jobs\test\workspace>start call pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt
e:\.jenkins\jobs\test\workspace>exit 0
Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information
Robot results publisher started...
-Parsing output xml:
Failed!
java.io.IOException: Error on line 195 of document file:///E:/Reports/Secretariat/output_xxx-20110818-142217.xml : XML document structures must start and end within the same entity. Nested exception: XML document structures must start and end within the same entity.
at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:71)
at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:44)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at hudson.plugins.robot.RobotParser.parse(RobotParser.java:39)
at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:235)
at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:258)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.post2(Build.java:161)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604)
at hudson.model.Run.run(Run.java:1400)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:175)
Build step 'Publish Robot Framework test results' changed build result to FAILURE
Finished: FAILURE
ISSUE SOLVED
Changed batch command from
start call pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt
to
pybot -A E:\Workspace\trunk\Suite\Secretariat\BatFile\Arguments\Arg_xxx_FF.txt
Unknown User (laurentb)
Hello,
I have a problem with error message display within the plugin.
My situation is :
- I use of the last version of the softwares (Robot, Ride, plug-in, Python)
- When a Robot test case fails I raise an error with a multi-line messages (to see different errors)
those multi-lines strings are created using '\r\n' for cariage return (I tried '\n' before but result was the same)
My problem :
- Error message does display in multi-line in Ride or in the Robot output, but when I look at it within the Jenkins plugin, it is all in a single line.
See screenshot bellow.
Does anyopne has an idea of the problem ?
Thanks,
Laurent
via jenkins plugin : (not nice, not carriage return)

via RIDE/ROBOT (nice : carriage return handled)
Unknown User (josesa)
Is there any way to Aggregate Donwstream Robot Reports the same way it does for JUnit reports?
My idea is to have in the compilation job where the "installable units" are generated which will trigger the sucession of robot executions (bringup -> smoke -> regression, ..) to have aggregated the robot results of all downstream jobs in order to evaluate the quality of the artifacts in a single place.
Unknown User (hiyizhiyu)
Hi,
by robot-plugin, we can get test result overview from "Robot Framework test results" per build, and it can show all failed case in "All failed testcases". And my question is that I can not get the log file of each failed case in "All failed testcases"? is there any links? how can I configure it? I can only get a short message like followings.
Unknown User (syncsynchalt)
The plugin works great for us, would it be possible to add a class to the "Robot Results" column td so that it can be styled differently?
Unknown User (josesa)
Is there any REST api planned for this plugin ?
Unknown User (toliveira85)
The plugin is failing if output.xml was created from several merged output.xml (using rebot --merge)
Unknown User (dabarma1)
I have the same problem. And give me the following failure:16531 Killed rebot -L DEBUG --nostatusrc --removekeywords wuks --removekeywords passed --report $WORKSPACE/report_merge.html --output $WORKSPACE/output_merge.xml --log $WORKSPACE/log_merge.html --xunit $WORKSPACE/xunitResults_merge.xml --merge $WORKSPACE/output.xml $WORKSPACE/output2nd.xml
Error merging
Do you found any solution?
Unknown User (omegawasp)
Is there any Environment variables for the result? Such as BUILD_NUMBER means the Build number that I can use this variable in Email notification plugin.
Now, I want to use Email-ext plugin to send Robot framework result in email content, like:
The automation result is XXX total | XX pass | XX failed.
The failed cases are XXX XXX XXX.
Unknown User (edsherwin)
Encountered this error in opening the report of Robot Framework. Thanks in advance. :D
Opening Robot Framework report failed
Unknown User (michel117)
Hello,
we use jenkins pipelines, and would like to integrate this plugin. We use it already for the normal jenkins job, but I wonder if someone is working on adapting to jenkinsfile/pipeline files.
Any comments?
cheers
Michel
Unknown User (dbubenik)
Looks like you can, but there was something broken with the snippet generator. In the snippet generator it's under "step: General Build Step".
I was able to publish robot results with the following snippet:
Unknown User (siimes)
Hi,
I've been trying to make automated Robot email reports through Jenkins and for that purpose I have used a Groovy-script. So far I've been successful on making pretty good looking reports in HTML format but haven't yet figured how to get the "warning/error" information attached to the failed test cases.
Has anyone else been tackling with this problem?
Unknown User (philosophy)
Hello
My Environment is below, The jenkins was installed on ubuntu, My PC(Windows7) as slave in jenkins. I using agnet by Java Web to via windows.
I was run windows batch command "pybot.bat --version" in slave. I got below issue:
E:\Jenkins\workspace\RF_Windows>pybot.bat --version
Robot Framework 3.0.2 (Python 2.7.14 on win32)
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE
Unknown User (tattoo)
The plugin expects you to have output.xml, log.html and report.html files existing in post-build step. {{pybot.bat --version}} naturally does not produce the files.
Unknown User (givadogabone)
Hi,
I installed this plugin with Jenkins on Linux.
Your step below do not work:
3.) Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
it's saying "not found".
Regards,
George
Unknown User (tattoo)
Hi Unknown User (givadogabone),
Please follow Robot Framework's instructions how to install and how to run the actual tool. This plugin operates on the various result files Robot Framework produces.
BR,
-TKa
Unknown User (arthur_chang)
Hi,
I installed Jenkins 2.150.3 and robot framework plugin 1.6.5, but there is no post build action "publish robot framework tests result". What should I do if I want to use robot plugin for Jenkins 2?
Best Regards,
Arthur
Unknown User (tattoo)
Hi Unknown User (arthur_chang) ,
Something has gone wrong with the installation of the plugin. Please re-install the plugin and restart Jenkins.
BR,
-TKa
Unknown User (arthur_chang)
Hi Unknown User (tattoo),
Thank you for your reply, I reinstall the plugin and restart Jenkins, it works now.
Thanks & Regards,
Arthur
Unknown User (almahajan)
Hello,
I have request to claim failed tests from Robot Framework tests reports in addition to claiming the failed job. i see this feature is available for Junit Plugin JUnit Plugin . Can we have this feature added as part of Robot Framework plugin as well?
Thanks,
-alok
Unknown User (tattoo)
Hi Unknown User (almahajan),
Please submit bugs or feature requests to the issue tracker. They won't otherwise be considered at all.
Unknown User (almahajan)
Unknown User (tattoo) Thanks, i tried searching if there's any open request for this but couldn't find anything related. So, created Feature request JENKINS-56862 - Getting issue details... STATUS
Regards,
~alok