Child pages
  • Test Results Analyzer Plugin
Skip to end of metadata
Go to start of metadata

Plugin Information

View Test Results Analyzer on the plugin site for more information.

A plugin that shows history of test execution results in a tabular format.

About

While executing our tests on Jenkins many of time test fails on particular builds. Now if you want to check that when a particular test-case, test-class or a test-package has failed you may need to go to each and every build report and check for the result.

This process is very cumbersome and may take a lot of time to get the report. The said plugin solves the said issue by showing the build result history of test-class, test-class and test-package in a tabular tree format. The plugin can be used enabling the "Publish junit results" or "Publish TestNG results"(in case of TestNG) feature of Jenkins.

It allows users to filter the results based on passed, failed and skipped status. 

Once installed you can just click on the "Test Results Analyzer" link on the left hand side of your job, as shown in the following image:

Following is a sample image of the report:


Graphs

We all know the importance of the graphical report for our test execution as it gives us a clear picture of the execution. The said plugin also supports generation of Graphs for the test execution from 0.2.0 version onwards.

Following charts are available:

- Line Charts

- Pie Charts

- Bar Charts

Following are the sample images of the generated charts:



Note: If you click on any point on the line chart it will generate a pie chart for said build/point.

Change Log

Version 0.3.1
  • Feature to search for a package, class or method name. Thanks to "Ayush Jain" for the changes
  • Added feature for loading icon to be shown while fetching and loading the report.
Version 0.3.0
  • Support for Downloading test-results in CSV format. Thanks to "Shahzad Bhatti" for the changes.
  • Added hyperlink to test results analyzer report for Junit reports. Thanks to "Omar Elabd" for the changes.
  • Added hyperlink to test results analyzer report for TestNG reports.
  • Added functionality to detect new Failures in in table. Added a new Column with an icon that is displayed when a new failure occurs. Thanks to "Ayush Jain" for the changes
  • Support for creating report for specified number of builds rather than fixed set. Thanks to "Shuo-Yang Wang" for the changes.
  • Added Global config for setting default chart options. Thanks to "hoff121324" for the changes.
  • On load show tabular report and charts and adding options menu for changing the default options. Thanks to "hoff121324" for the changes.
  • Show runtime graph. Thanks to "hoff121324" for the changes.
  • Added support for setting custom status names in tabular report in Global config.
  • Added support for setting custom status colors in tabular report in Global Config. 
Version 0.2.3
  • Support for parsing TestNg results from the "TestNg results" plugin and using it for generating the report. Thanks to "Markus Krügwe" for the changes.
  • Changes to fallback font to "Sans Serif". Thanks to "Derek" for the changes.
  • Updated the index jelly shown in the plugin repository. Thanks to "Donald Woods" for the changes.
Version 0.2.2
  • Added option to display execution duration in build report by going to Advanced options -> Display Duration. Thanks to "Evgeny Himmelreich" for the changes.
  • Fixed issue related revers display of builds when user chooses limited build to see in the build report.
Version 0.2.1
  • JENKINS-29111 - Show packages/classes/methods in sorted order
  • JENKINS-27628 - Test results analyzer displays class as FAILED when a testcase is skipped
  • JENKINS-29186 - Pie chart title showing a wrong build number when the chart generated for a build from line chart
  • Changes done to UI look and feel for the table and charts. Thanks to "RICHARD Bruno" for the changes.
Version 0.2.0
  • Changes done to generate line, bar and pie chart for the report.
  • Allow user to generate a pie chart for a particular build when user clicks on a build in line chart.
  • User have the options to choose from line, bar, pie or all charts

76 Comments

  1. Please share some documentation or screenshot. It will be helpful to understand.

    1. Hi Arpit,

      I had updated the document. Please let me know in case you need any more info.

  2. I don't see "Test Results Analyzer" link, even if the JUnit test is reported in Jenkins.

    I used "Public JUnit test result report" in the post action.

    1. I am seeing the same issue. I have a free style software project and added 'Publish JUnit test result report` in my post actions and don't see any Test Results Analyzer link in the left menu of my job. Any ideas on what the issue may be?

      EDIT: Restarting Jenkins after installing the plugin fixed this issue.

  3. Just started using this plugin, and I like it!

    I have one issue with it though: the plugin seems to be visible only for users with administer access (i.e, other users don't see the link to 'Test Results Analyzer').
    I'm fairly new to Jenkins, so I don't know if this restriction is due to the plugin itself or to a global Jenkins setting. Can I make a modification so that non-admin users can see the link to the test results analyzer as well? (and if so: how?)

    Thanks,

    Erik

  4. Just got some feedback from Varun: the plugin was updated to version 0.1.2 to support visibility for non-admin users as well.
    I tested it, and it works perfectly - thank you Varun!

    Erik

  5. Is it only for java based project or can we use it for .net projects?

    1. Hi Arpit,

      First of all sorry for late response. I dont get notification of any comment of users.

      The plugin compeltly depend upon the data generated by the Junit results publisher plugin.

      In case you are using the said junit publisher plugin the Analyzer plugin should also work for you.

      - Varun

  6. This seems to work great with free-style software project, however been able to make it work with multi-configuration project is this a known limitation?

    Thanks,

    Louis.

  7. Hi Varun,

    Similar to Louis I am using *multi-configuration projects.  *Is support for this in the pipeline?

    Regards,

    Steven

  8. Hi Louis, Steven,

    Thanks for asking the query related to the multi-configuration project. I suppose the plugin may not work in the said case.

    I will look into this and add a support for it.

    - Varun

    1. I was able to get around the multi-config project problem (or something similar) by copying the artifacts to the upstream job. Then by viewing the Test Results from that parent job, everything shows up.

  9. Are there plans to add an export feature to the generated report?

  10. I have tried using the analyser on a build that uses GoogleTest on C++ code, and no results are generated and the Jenkins server stops working and requires a restart (on Windows).  The GoogleTest output is parsed successfully by the JUnit plug-in however (with less detail of course).

    I am also using it on test output from our own in-house developed test tool for system verification on an embedded target which outputs JUnit plug-in compatible XML (as defined here), and on this it works brilliantly.

    If there is any chance of making this work with our GoogleTest output I'd be most grateful, as the results we are getting from our system verification tool are invaluable.  I have attached an example of the GoogleTest XML that fails if it helps: test_detail.xml

  11. I've started using this plugin but for some reason it only ever has the most recent build in the table/graphs and doesn't take older builds test results into account (which makes the graphs not much use). Any ideas on why this might be happening?
    Running Jenkins ver. 1.620

    1. Can you try with the latest. Earlier the plugin was only supported for Junit. recently we added supported of TestNg results and it started working for many users who reported a similar problem as yours.

  12. Does this plugin work for Maven projects, rather than just freestyle ones?  I can see 'publish JUnit test result resport' in a freestyle project post-build actions, but not for Maven.  I tried just doing a build after I'd installed the plugin (and restarted Jenkins), but when I use the 'test results analyzer' link for my Maven project, nothing comes up.

    1. I have the same problem. I want to try this plugin, but it does not seem to work for Maven projects, as "Publish junit results" does not seem to be available :-(   and that seems a Jenkins bug that has been around for 5 years... https://issues.jenkins-ci.org/browse/JENKINS-9980

  13. I am sorry that this comment is a little bit unrelated to this plugin.

    I like this functionality.

    But, if lots for tests (above 1,000) and large history (over 1 year), this plugin might be not easy to use due to performance issue.

    It is too slow to view the result.

    Are there any open source tools similar to this plugin?

    For example, I want to store test results to database.

  14. I saw on the github repo a couple of commits that will add ability to export results to .csv. Any ETA on the release with that feature?

    thanks for a great plugin

    1. The feature to download CSV have been released. Please update the plugin to latest 0.3.1 version to use it. Thanks for using the plugin.

  15. This plugin is great!

    Hyperlinks to builds/test cases would be appreciated.

    Also, an option to name the jobs/builds according the the 'Display Name' of the project/build instead of the build number would be great!

    1.  I second the request to be able to alter the build number displayed. I'm even using the Build Name Setter Plugin to modify the build numbers of my job (1.0.34, 1.0.35, etc.), but the columns of the analyzer still show the native build number (64, 65, 66, etc.)

    2. The feature to to show Hyperlinks to  builds have been released. Please update the plugin to latest 0.3.1 version to use it. Thanks for using the plugin.

  16. Thanks for this great plugin. 

    What is the best way to make this plugin run automatically every time I visit the Dashboard?

    I am creating a test-dashboard in Jenkins and want to make the results available automatically when any user visits the dashboard page.

  17. Hi. I've changed the format of my junit xml report and then, the old test report table format is still in the database.
    How could I clean/remove results from old builds to not see there entries in the main table result?

    1. I've run into this issue as well: the table displays rows for all the unique junit results that have ever shown up in the build history, regardless of whether they have shown up in the last '# of builds' that the plugin is set to analyze.

      I was able to work-around this by deleting the old builds (and/or their artifacts), but having to do this was undesirable.

    2. Hi,

      At this point the plugin does not maintain its own table or storage for storing results.

      The plugin uses the results stored in the jenkins to parse and then show it on the UI.

      For now if you want, you can limit the no. of builds to be shown in the report

      1. Even if you limit the number of builds, the plugin must still be going through the entire build history each time it produces the table because test rows which are only from builds that are not in the table still show up in new tables.

  18. Where does this plugin get the datas from? I have a Result now, but there where some changes in the built and it still shows the old "TestSuite" even tho i cleared out the workspace xmls and even deleted the build. Is there a way to "refresh" / clear the Test Result Analyzer to a state where he starts fresh.

    Even if I am setting the Build Numbers to only show the last, it still shows me the old Build with N/A Data

    1. As I saidin earlier comments and in the description the plugin uses reports stored by the Jenkins Junit/TestNG results plugin.

      In case you change and delete the build information of a job and you are not able to see the results in the said junit/testng plugins then you should raise the issue to respective plugin owners.

      There is a plan to have in-built storage to be used by test-results-analyzer plugin but that may take some time to be release.

      - Varun

      1. That wasn't his concern. He wanted to stop seeing things that he was seeing, not start seeing ones that he wasn't seeing.

  19. I am trying to easily email, convert to HTML or even print the graphs.  Is there anyway to do this or any thought of adding that functionality in the future?  I know I can print screen but looking to send just the graphs and results to business line representatives and I don't want the sidebars on there.

    1. Hi,

      I had released a new plugin 0.3.4 where you can download the graphs as png images for emails.

      For results export there is only csv download that is available currently which you can use for sending out an email.

      - Varun

      1. Thanks very much for the update.  I don't see where the 0.3.4 version is?  I do see the latest version is 0.3.3, is there a release date or schedule?

        1. Its available now, please download. It normally takes 6-12hrs after the release for the plugin to be available publically.

  20. Hi,

    please, could you add the "description" tag of the test in the report ?

    I need also to generate the results in a file (pdf).

    Thanks

  21. The Pi chart shows the values inclorrectly.

    See the attached.
    The Line graph shows the correct results, but the pi chart doesn't represent the same. Looks like It shift the typeand represents another type.

    1. Just wanted to explain on the functionality of pie chart here. The pie chart in the said plugin case shows the overall result of the builds that ran. If all builds failed it will show 100% as failed.

      The above issue or usecase is a corner case where no tests were executed due to multiple reasons. How you represent a build that dont have a test-case execution may change for person-to-person and organization-to-organization. In the current implementation I represent the build as skipped. I can make changes to neglect it but that from my point of view may not be right.

      Please feel free to share your thoughts

      1. Hi Varun,

        Sure I think I misread the topic of the graph. And I don't think you should remove the skipped runs as well. That shows the full view of the rest runs so far.

        But I do have a though of the position of the pi chart. The line and bar charts sows the similar information. Therefore the they should be together and the pi chart should not be between them. Wouldn't it be better represented if the Pi chart moved to the top of the three charts or to the bottom of the three charts ?

      2. Hi Varun,

        Firstly, I would like to say that the plugin is really a good one. But in regards to "Pie Chart" it could be nice if the pie chart provides you the same result as the other two graphs.

        What do you say? Because I feel that pie chart would more needful in understanding the test results of the latest build. Having a global idea of the status of the builds is really a nice option. Perhaps one could be more interested in the test results rather than build status itself. And also, it would justify the name of the plugin "Test Results Analyzer plugin".

        What do you think?

  22. Hi Varun,

    Thanks for creating this! I am having the following issue: Immediately after installing the plugin everything worked fine. However after running the maven build job once after installing this plugin the chart comes up blank. I have attached a screenshot.

    I have confirmed that the build is successful and the Jenkins default JUnit reporting tool is able to display the results. Is there a log file in the plugin directory? How can I go about troubleshooting this?

    1. Are you selecting Test Results Analyzer from the Jenkins Project view? It only works from the Module view.

      I fell into this trap, even after reading the error message several times.

      Try selecting select Modules from the Project Menu, then select a Module from the list, and finally then select Test Results Analyser from the Module view.

      1. That was it! Thank you so much!

      2. Iain, can you please tell me how can i get to Modules menu? I do not have such in Jenkins. Why does this plugin appears in Project view but does not work ??

        thanks,

        Anna

        1. If you start at your Jenkins home page, you should see a list of projects. Simply select one of them to go to the project view. On the left side of the Project view there should be link for Modules, which takes you to the modules list.

          I don't know why the option is displayed on the Project view when it doesn't work, it could be a limitation of the plugin or Jenkins itself.

  23. Is it intentional behaviour where skipping all tests results in a pass?

    I have disabled some tests by ignoring them in JUnit, since they're incomplete and giving false results. However the parent class and package are reporting as passed. I would expect/hope the entire package to report as skipped if all the contained tests were disabled. 
    Is there a way to mark the containers as skipped?

    1. I've got the same problem on my project with this kind of JUnit result configuration :

      ...

      <testsuite name="7-_Test_Business" tests="0" skipped="2" errors="0" failures="0" time="0.012">

      <testcase time="0" classname="7-_Test_Business.7-1_OXE" name="7-1-1-1a_-_Initialize_SIP_session">

      <skipped/>

      </testcase>

      <testcase time="0" classname="7-_Test_Business.7-2_CUCM" name="7-2-1-6b_-_Finalize_SIP_session">

      <skipped/>

      </testcase>

      </testsuite>

       ...

      I would like to know if you ,or someone, have found a solution to share a status SKIPPED to a testsuite composed by only skipped testcase?

  24. In a future release it would be nice to be able to tag the Build Number with a Date and a Tag of some sort.  That way I could label the graph with the date of the build and our internal build version.

    Thanks for all the developers and admins do, this tool is very handy.

  25. Is there any option not to display methods like afterClass/beforeClass and so on? 

  26. HI , this is a great plugin.

    I have 2 nice to have features : 

    add statistics per test - for example after 20 runs the right column will be the summary - 16/20 or 12/20

    filter out stable tests - if a specific test passed all the time filter it out from the report - it will be easier to focus on the failures.

    Thanks , Mor

  27. Hi, i have a question. Can you help me ?

    when i  run "batch runs" in jenkins, when test finished, why i see nothing  by  "Test Results Analyzer" ?  Is the "Test Results Analyzer" not support batch runs ?

    The capture is below:
    Hope your answers, thx!

  28. I have removed some of my invalid test cases from the test suite , and they keep on appearing on the results. They Appear with "N/A"  for the result. How can we remove them  from the table?

    1. These test cases results may be present in one of your execution and hence they are shown in the report as not executed. 

      At this point there is no feature to skip them to be displayed

  29. Hi Everyone,

    I am really stuck with one of the issue. Any help from anyone would be very much helpful for me.

    Is there any way, That I can display my customized method name on this report?? 

    If yes, How do we implement these changes.?

    Thanks

    1. The results for them the junit plugin and rendered by the said plugin. At this point you cannot show customized method names in the plugin

  30. It seems that the plugin uses too much CPU while loading the results.
    I have hosted Jenkins on a windows server and when I try to view the results for a job with around 300 builds.
    The CPU reaches 100% usage and Jenkins becomes unresponsive.
    When I close the tab the same comes down to around 10-20%
    Any solutions?

  31. Hi Varun,

    Thanks for this plugin! I am a little bit confused with run time for each test. Would you mind to explain how the digit per test is calculated? (401.003 on my screenshot)? I do not see anything like this in testNG report.

    Also, is there a way to configure plugin to show times instead of Passed/Failed for a session or per all project? 

    1. First of all: really useful (and pretty :) ) plugin. Thanks :)

      @LL LL, Hello, I would guess you are seeing the same bug as me - or a related bug.

      I see that there is something funny when showing test run times in table: 2 bugs I think.

      I have one test suite in each junit xml results file.

      I'm using latest Test Results Analyser Plugin 0.3.4.

      1.

      I see that in the first column (for latest test run) that the run time shown for each test file and test suite is the same for all.

      The run time shown for all looks like the time for the first test suite.

      The run time shown for all looks matches the sum of time for each test in first test result file + a little extra.

      2. 

      After the first column the time shown for junit xml file looks correct (matches the sum of time for each test).

      But the time shown for each test suite is the same as the first test suite of that column (or build).

      1. Hello Varun,
        Thank you very much for this plugin!
        I'm using it for publishing test result converted to xUnit format. Table loads all required data, but unfortunatelly duration are displayed the same time for all tests in a test run. Here is attached xml file and screenshot with the results.
        Hello Varun,

        Thank you very much for this plugin!

        I'm using it for publishing test result converted to xUnit format. Table loads all required data, but unfortunatelly duration displays the same time for all tests in a test run. Here is attached xml file and screenshot with the results.

        Could you please clarify what is wrong with the xml file or with the durations?

        Thanks,
        Bohdan

  32. We have retries that show up as skips in the test results but they don't show in the table. Is that normal behavior?

  33. Hello Varun,

    I would like to use your plugin but cannot seem to get it working - getting message: "No build data retrieved. You may need to select a Module."

    After the plugin is installed - what are my next steps to get it working? 

     Is it necessary to enable ""Publish TestNG results" plugin?

  34. I'm trying to get this to work for my TestNG output but when I try to build, I get an empty set.  What am I missing?  There is no "module" for me to select, and I don't get that same message others did above. 

    I'm guessing I'm missing a config step somewhere.  My TestNg Results works correctly, and I have a trend, so I know results are being published, and archived successfully.

    1. plugin is not finding your test- reports xml file. it would be great if ur code is on github .

  35. Dear Varun,

    At the moment we're using your plug-in with jenkins for software testing - eventually hoping to advance to development as well. We have, however, run into a problem to which we haven't found any instruction for. In other words the Test Analyzer plugin always takes even the past test scripts into current reports regardless if the old scripts/test have been removed from SVN.

    Please advice - we're using this for the first time next week and we need detailed analytics without N/A / old test scripts that no longer ever are in our SVN. 

    Thank you in advance,

    Tom Kuisma

  36. Dear Varun,

    At the moment we're using your plug-in with jenkins for software testing - eventually hoping to advance to development as well. We have, however, run into a problem to which we haven't found any instruction for. In other words the Test Analyzer plugin always takes even the past test scripts into current reports regardless if the old scripts/test have been removed from SVN.

    Please advice - we're using this for the first time next week and we need detailed analytics without N/A / old test scripts that no longer ever are in our SVN. 

    Thank you in advance,

    Tom Kuisma

  37. Hi,

    The plugin has a problem when parsing results of multiple executions of the same test class in TestNG. On the class level, the plugin will only log the result of the last execution, while on the method level those of the first execution, resulting in confusion. It would also be paramount to have it show the parameters you're passing in each test. Do take this into account as it is a critical flaw.

    Regards,

    Stratos

  38. Dear Varun,

    Thank you for nice plugin.

    I am using it to export test results in CSV file, I want to know is it possible to add in exported file the date of builds instead of build number ? 

  39. Hi Varun,

    Thank u for the plugin. This is regard to the email sending of the test results. I would like to know whether there's a way to attach the results displaying of this plugin to a email notification after each build is executed? i'm glad if you can reply to this because we have looked for many options and nothing worked.

    Thank you.

    1. I have asked similar questions in the past and as far as I know, all you can do is export the graphs and email them.  I love this plugin but would be nice if there was a way to email the displayed results in a picture format instead of exporting and attaching the graphs.

  40. Thanks for creating the plugin its really very useful and comes in handy to check the build statues. However i do have a question:
    I see the build number being displayed as build serial numbers (auto incremented numbers 1...61.. etc). In the build i created the number to be a : <Branch>-<Build Number>, is there a way to display that as the build names

     

     

    Build names :

  41. Hello Varun,

    Sometimes test case shows passed/failed in test result analyzer report but sometimes it shows N/A. Can you please tell me the reason behind this...?

    Thanks in advance

  42. HI Varun,

    this plugin really sounds great and easy to use. i have used this plugin for the purpose of result analysis and this really helps in determining Go/No-Go for production push however i have a challenge infront of me i.e. is there any way to compare failures if any, in between previous build and current build results ? i mean if 3 TC failed in current build and 3 were failed in previous build, is there a way to compare if the failure were for the same TC or for different TC? if not, may i request you to incorporate this feature in your plugin? it will really be helpful.

  43. This seem great .

    Few thing that I would like to have :

    1. Link to Git so we can add things
    2. Filter test according to fail/pass
    3. Search 
  44. Curently it seems that the parent class and package are reported by Test results analyzer Jenkins plugin as passed even though if they are only composed by skipped testcases

    I would prefer the following behavior :

    - if a test suite is composed only of success and skipped test then its status must be success

    - if a test suite is composed of at least one test in error then its status must be error

    - if a test suite is composed of only skipped tests then its status must be skipped

    Do you know if it's possible to configure the Test results analyzer Jenkins plugin to do that ?

    The problem with the actual behavior is that if few testsuites are success even thought there are only composed of skipped, you cannot quickly detect the evolution if one of them newly containing a success test. To my mind it's a lack of information given to users of this plugin

  45. By default, I display the results of the last 10 builds. Unfortunately, I also get results with N/A which are 100 builds and more behind and no longer exist. Because of artifacts and references to the issue tracking system I cannot delete them. I would therefore be happy to have the option to suppress test results that are not listed in the visualized build series. This means that if the whole row is N/A, the whole row should be suppressed.

    Is there a way to request this as a feature or improvement?

Write a comment…