Due to some maintenance issues, this service has been switched in read-only mode, you can find more information about the why

and how to migrate your plugin documentation in this blogpost

Skip to end of metadata
Go to start of metadata

Plugin Information

View Performance Publisher on the plugin site for more information.

This plugin is up for adoption. Want to help improve this plugin? Click here to learn more!

This plugin generates global and trend reports for tests results analysis. Based on an open XML tests results format, the plugin parses the generated files and publish statistics, reports and analysis on the current health of the project.

Context

CAPS entreprise provides software developers with an easy access to manycore systems thanks to its flagship product HMPP™ (Heterogeneous Multicore Parallel Programming).

Based on C and FORTRAN directives, HMPP offers a high level abstraction of hybrid programming that fully leverages the computing power of stream processors without the complexity associated with GPU programming.
HMPP compiler integrates powerful data-parallel backends for NVIDIA CUDA and AMD CAL/IL that drastically reduce development time. The HMPP runtime ensures application deployment on multi-GPU systems. Software assets are kept independent from both hardware platforms and commercial software. While preserving portability and hardware interoperability, HMPP increases application performance and development productivity.

The HMPP product validation criteria are many and various (with regard to the supported languages, the behavior of the directives, in relation to the OS, with regard to performance,...). So, a validation infrastructure has been developed based on internal tools and hudson projects for the control and the pretty-print of reports.
In the following, we only describe what we call the performance publisher part. This part is dedicated to the pretty-print of the report resulting of the validation phase. The usual Hudson functionalities like the management of the projects, of the slaves, the automatic launch of the command, the notifications, etc. are not described in the present document.

Description

The PerfPublisher plug-in scans for xml tests reports defined in the administration panel and generates trend graphs, computes stats and underline regressions and modifications. The xml files must be generated by your own testing tool.

Features

The following features are provided by this plug-in:

  • Configuration of the plug-in
    • Multiple reports
    • Unstable threshold: Set the project as unstable when a number of failed test is reached.
    • Health reporting: Interpolate the project health in function of the percent of failed test.
  • Multiple analysis point : A test can publish multiple measures :
    • Status (success / fail / broken)
    • Compile time
    • Execution time
    • Performances
  • Build summary : showing the number of failed and successful tests, number of files and number of executed tests.

 

  • Several trend reports
    •  showing the evolution of failed, broken and successful tests and the evolution of three types of measures : Compile Time, Execution Time and Performances per build.


  • Global build report of the tests executions status.
    • Tests can be categorized ( per categories and/or xml-file )
  • Graphic view of the results
    • The tests can be characterized by their declared target.
  • Stats on results and their trend
    • (Number/Percent/Trend) of executed tests
    • (Number/Percent/Trend) of not executed tests
    • (Number/Percent/Trend) of successful tests
    • (Number/Percent/Trend) of failed tests

  • Multiple analysis points
    • (Number/Average/Trend) of compile time
    • (Number/Average/Trend) of execution time
    • (Number/Average/Trend) of performance

  • Tests history
    • (Number/Percent) of new tests
    • (Number/Percent) of deleted tests
    • (Number/Percent) of execution changed status tests
    • (Number/Percent) of success changed status tests

  • Possibility to perform reports diff.
    • Manual export of a TXT diff summary of the current build ( to use with your favorite diff application)
    • Integrated diff between none concurrent builds ( up to 3 builds )

  • List of the executed tests
    **Can be sorted by target, name, status, compile time, execution time, performance.

  • List of the success changed status tests

  • Each test has its own description page
    • Execution and result trend
    • Status
    • Measures
    • Platform description
    • Unlimited number of log (error logs compile log, execution log... )
    • Execution parameters

DTD, XML FILES

Any xml report file, must be validated by the following DTD : DTD report.

How-to generate your reports

Each "build" can contain multiple Xml reports, representing a part of the repository of tests you have executed on your project. The results will be included in the computation of statistics as a comprehensive analysis but also more specifically, as sub-reports analysis (by category and by name of file).

The contents of XML files has been developed to be as generic as possible. The aim of this plugin is to integrate all your test results and to generate an efficient quality metric defining the current health of your project. For this metric to be effective, we have been providing a report format adapted to a large number of types of tests. (We should say, "we tried to...")

A report is structured around the following key elements:

<report name="{REPORT_NAME}" categ="{CATEGORY_NAME}">
  <test name="{TEST_NAME}" executed="{EXECUTED_STATUS}">
    <result>
     <success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
    </result>
  </test>
</report>

Warning

These elements are mandatory

Useful Information

The plugin was developed with a strong constraint: "a number of tests well above 5000 per build".

From a point of view a little bit more technical, these tests are considered as members of a file and more generally, of a category. However, each test is independent of others. It possess its own definition, and especially its own execution context. However, we mention that a forthcoming features, will provide case simplifications of writing and reporting through mechanisms of factorizations.

You can provide a large amount of information on each test in order to characterize them.
To go further more into the details of all available tags, I suggest you view the reports generated by the "xml-generator" application.

Of course a test can have a complete description. And furthermore, a test ban be considered as "boken" with the EXECUTION_STATUS attribute (yes|no). A broken test, is test that you know should exists but you've have disabled it.

[...]
 <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>&lt;![CDATA[This is the description of the test number 0 member of the 1 file.]]</description>
[...]

But it can also be "targeted". That is, you can set it a target of testing, one or more keywords describing the purpose of the test. For example, in the case of a test solution for a web application, you can set the targets tested by languages

  • CSS
  • HTML
  • JAVASCRIPT
  • ....
    or rather preferred functional approach analysis using the following targets:
  • SECURITY
  • HMI
  • EMAILS
  • ...
    [...]
      <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
        <description>{TEST_DESCRIPTION}</description>
        <targets>
          <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
        </targets>
    [...]
    

    Useful Information

    A test can have multiple targets. The attribute THREADED_STATE(true|false) has been integrated for language like C, FORTRAN, which can be threaded

Let's deal now with the context of the implementation of a test. Experiments run on your machine, which can be dedicated, and which can intervene heavily on results such as performances or just for a test repository adapted to the notion of integration. You have the ability to describe this context using the following tags.

[...]
  <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>{TEST_DESCRIPTION}</description>
    <targets>
      <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
    </targets>
    <platform name="{PLATFORM_NAME}">
      <os>
        <type>&lt;![CDATA[{OS_FULLNAME}]]&gt;</type>
        <name>&lt;![CDATA[{OS_NAME}]]&gt;</name>
        <version>&lt;![CDATA[{OS_VERSION}]]&gt;</version>
        <distribution>&lt;![CDATA[{OS_FULL_VERSION}]]&gt;</distribution>
      </os>
      <processor arch="{PROCESSOR_ARCHITECTURE}">
        <frequency unit="{FREQ_UNIT}" cpufreq="{CPU_FREQ}" />
      </processor>
      <hardware>&lt;![CDATA[{HARDWARE_INCLUDED}]]&gt;</hardware>
      <compiler name="{COMPILER_NAME}" version="{COMPILER_VERSION}" path="{COMPILER_PATH}" />
      <environment>{OTHER}</environment>
    </platform>
[...]

Useful Information

  • You can only have one platform description per test.
  • Only one os per test
  • Only one processor architecture
  • Multiple processor frequency per processor architecture
  • Multiple hardwares
  • Multiple compilers
  • One environment description

Useful Information

ELEMENT

UNIQUE

MANDATORY

DESCRIPTION

PLATFORM_NAME

YES

NO

The name of the platform.

OS_FULLNAME

NO

NO

The name of the os. (ex. : Linux-2.6.26-2-amd64-x86_64-with-glibc2.3.2)

OS_NAME

NO

NO

The name of the os. (ex. : Linux)

OS_VERSION

NO

NO

The version of the os (ex. : 2.6.26-2-amd64).

OS_FULL_VERSION

NO

NO

The full name of the version (ex. : Linux-2.6.26-2-amd64-x86_64-with-debian-5.0.3).

PROCESSOR_ARCHITECTURE

NO

NO

The name of the architecture of the test platform (ex. : x86_64).

FREQ_UNIT

NO

NO

The unit of the value representing the frequency unit of the cpu (ex. : MHz).

CPU_FREQ

NO

NO

The frequency value of your cpu (ex. : 2667.000).

HARDWARE_INCLUDED

NO

NO

A description of an included hardware in your test platform (ex. : nVidia Corporation GT200 - Tesla C1060 / Tesla S1070 (rev a1)).

COMPILER_NAME

NO

NO

The name of a compiler (ex. : ifort).

COMPILER_VERSION

NO

NO

The version number of the compiler (ex. 11.0 20090131).

COMPILER_PATH

NO

NO

The path to you compiler binary (ex. : .).

OTHER

NO

NO

Any other libs or environment data you want to include in the test report.

Let us focus now on the representation of test results. A test can have multiple output data. A simple test may called "boolean". That is, it just indicate whether or not the test succeed (or failed).
Coupled with this, we can provide, the percentage of success in the case of a test failed. For example, a test can fail at 90% of its execution status, representing a partial error, certainly important but partial.
It is also important to highlight the tests that have failed due to a timeout. You have the option to specify it.

In addition to this type of result, a test can provide important informations such that its compilation time, execution time or performance that it could have measured.
It is therefore important to trace this information. The format adopted proposes xml tags for the representation of these data. But event you want to save the values, for a reason on another one, you perhaps only wants few of the metrics to be included in the global statistics analysis. To include or not a measure in the stats, you can specify the IS_RELEVANT tag.

[...]
  <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>{TEST_DESCRIPTION}</description>
    <targets>
      <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
    </targets>
    <platform name="{PLATFORM_NAME}">
      <os>
        <type>&lt;![CDATA[{OS_FULLNAME}]]&gt;</type>
        <name>&lt;![CDATA[{OS_NAME}]]&gt;</name>
        <version>&lt;![CDATA[{OS_VERSION}]]&gt;</version>
        <distribution>&lt;![CDATA[{OS_FULL_VERSION}]]&gt;</distribution>
      </os>
      <processor arch="{PROCESSOR_ARCHITECTURE}">
        <frequency unit="{FREQ_UNIT}" cpufreq="{CPU_FREQ}" />
      </processor>
      <hardware>&lt;![CDATA[{HARDWARE_INCLUDED}]]&gt;</hardware>
      <compiler name="{COMPILER_NAME}" version="{COMPILER_VERSION}" path="{COMPILER_PATH}" />
      <environment>{OTHER}</environment>
    </platform>
    <result>

      <success passed="{SUCCESS_STATUS}" state="{SUCCESS_PERCENT}" hasTimedOut="{TIMEDOUT_STATUS}" />

      <compiletime unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <performance unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <executiontime unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <metrics>
        <{CUSTOM_METRIC_ID} unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />
      </metrics>

      <errorlog><![CDATA[{ERROR_LOG}]]></errorlog>

      <log name="{LOG_NAME}"><![CDATA[{LOG}]]></log>

    </result>
[...]

Useful Information

  • A test can only have one result tag.
  • A result can only contains one compile time, execution time and performance.
  • A result can only contains one errorlog but it can have multiple logs.

Useful Information

TAG

VALUES

DESCRIPTION

SUCCESS_STATUS

yes,no

yes:the test is successful.

SUCCESS_PERCENT

0-100

the percent of success (100=passed test).

TIMEDOUT_STATUS

true,false

indicates if the test has failed due to a time out

UNIT

String

The unit of the metric

MEASURE

Double

The value of the metric

IS_RELEVANT

true,false

indicates if the metric should be included in the global stats computation

ERROR_LOG

String

The failure log

LOG_NAME

String

The name of any other log you want to include in the report

LOG

String

The content of the log you want to include.

CUSTOM_METRIC_ID

String

The ID of custom metric represented as string

XML-GENERATOR

The tool xml-generator (xml-generator.jar) generates exemples of valid reports.

xml-generator usage

Usage : java -jar xml-generator.jar (number of file to generate) (number of test per file)
Exemple : java -jar xml-generator.jar 10 600

  • If you want to try the plugin with this tool :
    • Create an empty project (freestyle or multiconfig)
    • Execute a first build (it will create the project workspace directory)
    • Copy xml-generator.jar in the workspace directory of your project
    • Activate PerfPublisher plugin for your project (tick the option : Activate PerfPublisher for this project) and configure it to parse *.xml files
    • Add a build step to your project containing the following commands :
      On Linux :
      rm -f ${WORKSPACE}/*.xml
      java -jar ${WORKSPACE}/xml-generator.jar 5 200
      
      On Windows (thx to Jéssica Yoshie Kussuda)
      del ${WORKSPACE}\*.xml
      java -jar ${WORKSPACE}\xml-generator.jar 5 200
      
    • Execute a build
  • This tool generates xml random reports. It should respects the following random ratio :
    • 4 targets (C, JAVA, PHP, FORTRAN) equiprobables
    • 80% of executed test
    • 70% of passed test
    • On the 30% of failed test, 20% are timedout
    • 30% of compile time, execution time and performance measures are considered as relevants (included in the stats)
    • Up to 5 auxilliary logs

RELEASES AND UPDATES

The current release is available in the download section. This plug-in is developed and maintained by Georges Bossert. Please use the Hudson mailing lists or issue tracker to ask questions, create feature request or bug reports.

This plugin is provided as it is. CAPS entreprise provides no warranty on the correction of errors or its support.

Changelog

  • What's new in 7.98
    • New details Passed and Failed tests pages
    • Addind trend analysis for the last 5 builds
    • [BETA] Possibility to specify your own metrics
    • More efficient Categories and Files sub-reports.
    • A bit of i18n
  • What's new in 7.97
    • Error on release process has transformed this version as an unstable one. Should not be used !
  • What's new in 7.96
    • Update Matrix support
    • Resolve a jelly duplicated code section
    • Resolve few bugs
    • Optimize loading and memory usage
    • Offers a filter option in the build diff
    • Add links in the build diff to directly access tests results
    • Add a "regression alarm" on the build main page.
  • What's new in 7.95
    • Optimizes FreeStyle Project analysis
    • Reduces "New Executed Tests" statistics computation time
    • Offers direct links between Diff Display and associated results
    • Add a filter option to the diff display
    • Supports Multi-Configuration projects
      • Offers a "Global test report" including build trend for each combinations
      • Aggregates axes results of a build in a summary report
    • Rename menu links in function of reports
    • Prepare the support of official Test Result plugins
    • Update source code annotations and javadoc comments
    • Prepare the support of multi-languages via associated Plugin

19 Comments

  1. Unknown User (jessicayk)

    Remembering that the command

    rm -f ${WORKSPACE}/*.xml

    does not work on Windows.

    It's necessary to do

    del <the path with bars like this ..\..\..>*.xml

    =)

  2. Unknown User (xacocax)

    Would it be possible to change this plugin to support customizable performance metrics?

    For example, we're using Hudson to run tests on 3D games, and we'd like to be able to keep track of startup time, shutdown time, frame rate, and at least 4 or 5 different memory numbers. This plugin looked very promising, but only has 3 performance metrics (the names of which are not customizable).

  3. Unknown User (fdeweerdt@cloudmark.com)

    Hi,

    I'm getting an error with a multi-conf project:

    build702 $ /bin/sh -xe /tmp/hudson4780389519265024527.sh
    + java -jar /home/hudson/workspace/Trunk-multi/label/build702/xml-generator.jar 5 200
    Execute PerfPublisher Hudson Plugin report generator
    Any questions : http://wiki.jenkins-ci.org/display/HUDSON/PerfPublisher+Plugin
    Starting xml report generation...
    Number of file : 5
    Number of test per file : 200

    • Report Report-0.xml has been generated.
    • Report Report-1.xml has been generated.
    • Report Report-2.xml has been generated.
    • Report Report-3.xml has been generated.
    • Report Report-4.xml has been generated.
      ERROR: Publisher hudson.plugins.PerfPublisher.PerfPublisherPublisher aborted due to exception
      /home/hudson/workspace/Trunk-multi/label/build702 does not exist.
      at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:474)
      at hudson.plugins.PerfPublisher.PerfPublisherPublisher.perform(PerfPublisherPublisher.java:180)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
      at hudson.model.Build$RunnerImpl.post2(Build.java:158)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
      at hudson.model.Run.run(Run.java:1281)
      at hudson.matrix.MatrixRun.run(MatrixRun.java:130)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:129)

    I specified *.xml in the Reports to analyze textbox. The /home/hudson/workspace/Trunk-multi/label/build702 directory does exist on the said machine.

    Thanks,
    Frederik

    1. Unknown User (fdeweerdt@cloudmark.com)

      FWIW, I've tried the proposed plugin fix for the Windows/Linux slave issue problem. It solved the problem for me.

      Thanks,

      Frederik

      1. Unknown User (jfaenomoto)

        I'm running Jenkins on a master/slaves structure and I've got the same problem as you reported when running PerfPublisher on a slave. Recreating the path in master and copying all xml's solves the problem, which means the problem is related to where pĺugin gets its files. Is your fix avaiable for future versions? I'm running version 7.97.

        []'s

        1. Unknown User (fdeweerdt@cloudmark.com)

          There's a plugin with a fix that the author of the plugin posted here:

          https://issues.jenkins-ci.org/browse/JENKINS-6652

          (see Attachment)

          Installing this solved the issue for me.

  4. Unknown User (jl.pinardon)

    Dear all,

    Sorry for a perhaps dummy question, but I have not cleary understood how the XML file are filled.

    • As Hudons admins, should we create it with placeholders (e.g. {MEASURES}) which will be filled by the plugin
    • Or should we create a job calling a script to generate these files and replace token {MEASURE} by the true value for each job?
    • Also, I don't understand what's the signification of the "Command Line" elements, and why/how using it

    Thanks for your help.
    Best Regards
    J.L.P.

  5. Unknown User (jkahrs)

    Hello,

    I have just started working with the PerfPublisher plugin and just when I started to like its features, I noticed something ugly that should be easy to change for the implementor of the plugin. Test cases that failed are displayed with an intense red background so that the foreground text is hard to read. It is probably a good idea to change the background color to #FFC2C9. This is the same color used by CruiseControl to indicate errors. With this background color, the foreground color is much easier to read.

    Thanks for paying attention.

  6. Unknown User (avityuk)

    Very good plugin, we use it for publishing our test results.
    Only we have an issue with static icon urls. On web page there is "http://localhost:8080/plugin/PerfPublisher/icons/bullet_orange.png" url, which is not available. But if I downcase "perfpublisher" in url, it works fine.

  7. Unknown User (anguyen@cloudmark.com)

    Hi,

    simple question :  is it possible to specify multiple reports inside the same file?

    Thanks in advance for your reply.

  8. Unknown User (dtka)

    I have a weird issue. The job configuration page shows the checkbox option "Activate PerfPublisher for this project" twice.

    Any ideas why?

    1. Unknown User (jfaenomoto)

      I've got this issue too, using both latest versions of Jenkins (1.446) and PerfPublisher (7.97). Not only on job configuration, but generated reports as well.

      Didn't run for a solution, but got source code and tried some analysis, but got no result.

  9. Unknown User (teaspring)

    hi,

    I follow the test result xml format to produce a dummy result xml(as below) file to have a try.But it seems not be processed at all.After the build, the PerfPublisher draw a trend graph with test case count of 0 for every build.

    <report name="PerfPublisher_VDI01" categ="Frontend">
      <test name="Save Logic - RICSource/SingleRICOutput" executed="yes">
         <result>
              <success passed="yes" state="100" hasTimedOut="false"></success>
          </result>
      </test>
    </report>

    Then I use the recommended XML-Generator to produce an example test result xml file and it fails either. I did not update anything in the Xml-Generator generated file as it looks fine. 

    I listed the complete directory + file name in its configuration "Reports to Analysis".

    Any ideas? 

    1. Unknown User (teaspring)

      Further, I do not text any Metric Configuration. What to text for it?could anybody present an example?

    2. Unknown User (teaspring)

      I installed v7.97, and encounter the issue that the Customized Metric Configuration can not be saved in job config. It is said the later version fix it. From google I see the latest version should be 7.99(may be later), why the Jenkins Plugin manage centre does not recommend it?

  10. Unknown User (foxie)

    Please release the latest version 8.01 so that it will be available in the Jenkins update site (http://updates.jenkins-ci.org/download/plugins/) and can be updated through Jenkins Plugin Manager!

  11. Unknown User (foxie)

    Very nice plugin, thanks. I have one improvement need though: When viewing the test case metrics (select a build -> Build test report -> select one test case to view the details) it would be nice to see the actual performance values of each build on the graph. It would make it much more easier to compare the performance between releases.

    1. Unknown User (mlukasova)

      I agree with Foxie R, it is now really hard to compare two metrics. And i also think there is a bug with own metrics, when i have 6 owns, but not every xml gives me values for each of them, it shows wrong in test report columns (wrong columns). I also cannot set units, nothing appears in grapf, but in XML there is (for example) unit=\"MHz\". But nothing appears.
      Could someone tell me if i do something wrong or you have same problem?
      And does someone still work on this plugin? Is there some forum where i can post a bug?

  12. Unknown User (bazyli)

    Hello, I upgraded my jenkins and what happened is that some of the plugin components are displayed as plain text instead of html. Did such error occurred to someone ? Any idea how to fix that ? Thanks.