Jenkins
Space shortcuts
Child pages
  • Benchmark Plugin
Skip to end of metadata
Go to start of metadata

Plugin Information

View Benchmark on the plugin site for more information.



Benchmark plugin to compare boolean and numerical results over the history of builds.

About

This plug-in adds a post-build capability.
This plug-in purpose is to benchmark over all of a job builds, boolean and numerical results through tables and graphs.
This plug-in has also a mechanism of thresholds over numerical values to control a build success or failure.

This plug-in is also designed to load any type of JSON or XML result file format.
This plug-in works best in combination with an automated notification plug-in.
This plug-in works also as a result viewer for the 'jUnit Plug-in' and the 'xUnit Plug-in'.

This plug-in is accessible as a Jenkins pipeline component.


The documentation on how to use the plugin is located inside the Github repository.


For the plug-in one slide features, click HERE.

If you are interested in why this plug-in was created, please find the original proposal HERE.

Simple mode - Two steps use

The Benchmark plug-in in its simplest form works as a result viewer for the 'jUnit Plug-in' and the 'xUnit Plug-in'. 

  1. Add the benchmark plugin after a xUnit or jUnit post build step in the job configuration,
  2. Run the build.

The icon to access the plug-in result viewer should appear on the job main window.

Standard mode - Use registered schema

Here are the links to the standard schema:

To use one, just select it in the dropdown menu of the plug-in configuration. The necessary schema format (JSON or XML) will be automatically detected. 

Advanced mode - Create your own schema

Use the manual to build a custom schema: HERE

A custom schema can be either present in the workspace or added to a text field during the job configuration.

Changelog

1.0.8

Fixed issue where values of String parameters appeared as N/A.

1.0.7

Removed hyphens from the main tables to allow numerical sorting.
Removed text versions (true/false) of Boolean results from the raw table to allow numerical sorting.
Adjusted left the Group/Result/Unit names for readability and prevent the awkward display.
Fixed a bug that occurred when changing schema content where old parameters ended as results and crashed the display of the main tables.

If the table display is frozen due to this bug, the workaround is:

  • Update the plugin,
  • Go where the job is stored on the master node [requires access rights],
  • And, delete the file 'BenchmarkCondensed.json' in the job folder.

The file 'BenchmarkCondensed.json' will be recomposed during the next build.
Warning: Any result from past builds, absent in the new result file(s) will not be considered.

1.0.6

Following fixes by magik6k (Thank you).
Fixed issue with pipeline support.
Improved failure mechanism.
Added link inside README to a page on how to write the plugin script for pipeline.

1.0.5

Issues for last 3 updates identified by Shay Gal-on (Thank you)
Fixed issue with additional thresholds.
Fixed issue with result display.
Updated help text.

1.0.4

Fixed issue with multi-file processing.
Fixed issue with log content.

1.0.3

Fixed issue with ResultFull and ParameterFull for JSON.
Fixed issue with merge system.
Fixed issue with log content.

1.0.2

Switched loading of schema file from File to String; to InputStream to String.
Fixed issue preventing the link between condensed table and individual result pages.

1.0.1

First official release.

0.8.4

Initial commit as OpenSource under MIT license.
Added pipeline support.
Added Jenkinsfile for the plugin to be built by the Jenkins CI Infrastructure and check pull requests.
Activated SCM.
Fixed Findbugs warnings.
Added Findbugs exclude filter for specific warnings (see /src/findbugs/excludesFilter.xml for details).


30 Comments

  1. Shay Gal-On

    Is it possible to provide an example output file for the json?

    The simplest file that an application can create that will result in the plugin parsing it and providing output.

    e.g. how to convert something along the lines of:

    my_benchmark_metric_1,57

    my_benchmark_metric_2,32

     

    Into something for the plugin to consume. 

    In fact, if it could consume a simple csv that would be great!

    1. Daniel Mercier

      Hi Shay,

      For the result file:

      {
	"my_benchmark_metric_1" : 57,
	"my_benchmark_metric_2" : 32
}

      the schema should be:

      {
    "description": "Simplest result",
    "failure": { "value": true },
    "type": "object",
    "properties": {
		"my_benchmark_metric_1": {"type": "resultFull"},
		"my_benchmark_metric_2": {"type": "resultFull"}
	}
}

      For all-in-one "key":"value", there are two JSON specific types called resultFull and parameterFull.

      Cheers,

      Daniel

      1. Shay Gal-On

        Looks simple enough, but when selecting "custom" for schema, there is no way to feed in that custom schema. 

        What am I missing?

  2. Shay Gal-On

    NVM. Found it (smile)

  3. Shay Gal-On

    Still not working (sad)

    Created a simple job that does that:

    echo "{ 'throughput' : 10, 'latency' : 0 }" > tracking.json

    Pick it up on postbuild with custom schema:

    {
    "description": "Performance tracking",
    "failure": { "value": true },
    "type": "object",
    "properties": {
    "throughput": {"type": "resultFull"},
    "latency": {"type": "resultFull"}
    }
    }

     

    And when run I see:

    Benchmark plugin - Collection of results started.
File pattern: tracking.json
Number of results:          0
Number of parameters:       0
Number of added thresholds: 0
Results does not contain numerals.

     

    1. Daniel Mercier

      Your JSON content is incorrect. Use instead:

       echo { "throughput" : 10, "latency" : 0 } > tracking.json

      1. Shay Gal-On

        Same thing.

        From the workspace in the machine:

        $ cat tracking.json
        { "throughput" : 2, "latency" : 0 }

        Same output from the benchmark plugin

        1. Daniel Mercier

          Hi Shay,

          You found a bug. I identified two more during the fixing. It should work with version 1.0.3. The version was pushed to Jenkins Artifactory but it will take some time to become available on Jenkins. It will appear as an update.

          Thank you for finding the bug.

          Daniel

          1. Shay Gal-On

            Users are the worst thing that can happen to software aren't they... (smile) 

            1. Daniel Mercier

              Sorry mate,

              I have to disagree. I had to expedite originally this project because of another one; and so, I am glad that you took the time and had the patience to help me better the plugin.

              I am not sure I will be able to introduce support for two column CSV files anytime soon but I like the idea. If you get the chance, please log the improvement on Github.

              Let me know if you face any other issue or have any question.

              Cheers,

              Daniel

              1. Shay Gal-On

                gladly, though you will have to enable issues on your github project if you want anyone to log an issue...

                1. Daniel Mercier

                  Fair point. It is active now.

                  Thanks,

                  Daniel

  4. Shay Gal-On

    How do I get "groups"? Not clear from the docs what needs to be added.

    Thanks, Shay

    1. Daniel Mercier

      Hi Shay,

      A 'group' is simply a nesting object or nesting array declared as type 'object' or 'array' in the schema. It follows/records the JSON tree. There can be any depths of sub-groups. A group is named using the nesting JSON key but the name can also be customized using a nested key:value declared in the schema as 'type:name'.

      Note: There can be an issue with the display of results if the depth (number of subgroups) is very high or the group keys/names are very long (will also depend on the screen size).  

      Regards,

      Daniel

    2. Daniel Mercier

      Here is a modified version of your example (Windows batch):

      for /l %%x in (1, 1, 3) do (
   mkdir run%%x
   pushd run%%x
   echo {"test":{"T" : %%x, "L" : %%x}} > tracking.json
   popd
)

       

      Inside the benchmark plugin configurations:

      Result file:

      run*/tracking.json

      Custom schema:

      {
	"description": "Performance tracking",
	"failure": {"value": true},
	"type": "object",
	"properties": {
		"test": {
    		"type":"object",
 			"properties":{
	  			"T": {"type": "resultFull"},
				"L": {"type": "resultFull"}
  			}
		}
	}
}

       

      You should now get a group named 'test' in the result tables.

      Regards,

      Daniel

      1. Shay Gal-On

        Is it possible to make the group name non static? 

        Use case is to separate tests that run under different conditions. Think parametrized build, you want to group results with specific parameters together as they are not directly comparable... 

        1. Daniel Mercier

          Hi Shay,

          Do you have an example in mind ?

          For separate groups of tests with different conditions in the same result file, I would use something like:

          {
	"group_1": {
		"time": "5", 	
		"L": 12
	},
	"group_2": {
		"time": "10",	
		"L": 7
	}
}

          with schema:

          {
    "description": "Performance tracking",
    "failure": {"value": true},
    "type": "object",
    "properties": {
        "group_1": {
            "type":"object",
            "properties":{
                "time": {"type": "parameterFull"},
                "L": {"type": "resultFull"}
            }
        },
        "group_2": {
            "type":"object",
            "properties":{
                "time": {"type": "parameterFull"},
                "L": {"type": "resultFull"}
            }
        }
    }
}

          Note: Parameters can be grouped and still be attached to the result(s) next to the parameter group.

          Daniel

          1. Shay Gal-On

            I would like something where the test run can have:

            {
                "$PARAM": {
                    "time": "5",   
                    "L": 12
                }
            }

            PARAM is a build parameter, and I don't want to have a different schema for every possible value of that parameter.

            In the simplest case, think of grouping by the node the test is executing on, or by the compiler version used, or dataset used. You don't want a new job every time you make a change, and similarly you don't want to edit your schema. Is it possible to do something along the lines of:

            {
                "group_by": {
                    "group_name": "$PARAM",   
                    "L": 12
                }
            }
            Where param can be any string that comes from parametrized build. That way the schema does not need to change every time you want a new group. Instead of having a group called group_by it will take the name from the value of the group_name parameter. 

             

            1. Daniel Mercier

              Hi Shay,

              Sounds interesting, very dynamic and especially useful for filtering purpose.

              On the down side, results will become disconnected from one build to another because group path changed. File paths and group paths are used to identify and connect results between builds. This would also seriously affect the advantage of the result page (click a result line on one of the tables) by preventing the correlation between parameter changes and result changes.

              If this feature is really important to you, it can still be roughly implemented as described in your message with the current version of the plugin using one of the two approaches:

              • Generate filenames using the parameter value and thus moving the parameter embedding from group name to file name. The wildcard system will adjust to the custom filenames but it will create the disconnection mentioned above,
              • Or set it in stone by declaring each predicted parameter value in the schema as a group name with similar disconnections.

              Honestly, I don't think we should implement it like this. However, I can see an equivalent that could fit (let me know). It is something I considered before; but decided not to implement because of the overhead and because I was pressed by time. 

              The idea was to leverage 'array' to create N-dimension results. It would allow the display of 'result vs parameter vs build' instead of the current 'result vs build'. At the moment, an 'array' is considered as the holder of multiple results with similar format and uses the index as group name to identify/differentiate results. This behavior could be changed to hold the N dimensions of a result.

              I will look into it but I have no ETA.

              I thank you in advance for your understanding and continued interest in the plugin,

              Daniel

              1. Shay Gal-On

                Actually tried the parameter as filename, but the current layout makes that quite unwieldy (sad)  also the benefit of grouping is not there.

                Currently using exactly a schema that encodes lots of parameter values, but it seems counter intuitive and likely to end up in missing results because of using a different parameter value.

                I like the idea of using arrays to represent this, and you can then filter on specific values on the array as well, which would make perfect sense. Understood that this would be a more significant effort, was really trying to figure out if this can be achieved by some smart schema but seems like a limitation of current implementation. 

                Thanks again for your responsiveness!

                1. Shay Gal-On

                  what does parameterFull do, and where does it show up in results

                  1. Daniel Mercier

                    If you recall above:

                    {
    "description": "Performance tracking",
    "failure": {"value": true},
    "type": "object",
    "properties": {
        "group_1": {
            "type":"object",
            "properties":{
                "time": {"type": "parameterFull"},
                "L": {"type": "resultFull"}
            }
        },
        "group_2": {
            "type":"object",
            "properties":{
                "time": {"type": "parameterFull"},
                "L": {"type": "resultFull"}
            }
        }
    }
}

                    Two results called  "L" will appear under group_1 and group_2.

                    To visualize the parameters, access a result page by clicking on a result line inside one of the tables.

                    • At the top is the graph,
                    • Followed by the list of values. Each build/value can be selected by clicking on a value. The selected build/value is grayed-out/darker color.
                    • Below are the condensed information for the set of build values,
                    • Below for the 'selected' build is its value, status and any failure messages triggered by associated thresholds, 
                    • Finally, the parameters attached to that result/build/value. This is where you will find the parameter 'time'.

                    Regards,

                    Daniel

  5. Rajeev Gadgil

    I have schema like

    {
    "description": "Performance tracking",
    "failure": {"value": true},
    "type": "object",
    "properties": {
    "G1": {
    "type":"object",
    "properties":{
    "T": {"type": "parameterFull"},
    "g": {"type": "resultFull"}
    }
    },
    "G2": {
    "type":"object",
    "properties":{
    "T": {"type": "parameterFull"},
    "g": {"type": "resultFull"}
    }
    }
    }
    }

    my json results file is:

    {"G1":{"T" : 180, "g" : 946}}

     

    I get the following error

     

    Benchmark plugin - Collection of results started.
File pattern: tracking.json
ERROR: Build step failed with exception
java.lang.ClassCastException: org.jenkinsci.plugins.benchmark.condensed.StringCondensed cannot be cast to org.jenkinsci.plugins.benchmark.condensed.DoubleCondensed
	at org.jenkinsci.plugins.benchmark.parsers.MapperBase.checkThresholds(MapperBase.java:307)
	at org.jenkinsci.plugins.benchmark.core.BenchmarkPublisher.perform(BenchmarkPublisher.java:200)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
	at hudson.model.Run.execute(Run.java:1762)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)
Build step 'Benchmark results' marked build as failure
Finished: FAILURE

     

    Any insights on this?

    1. Daniel Mercier

      Hi Rajeev,

      I used the information you provided and ran the execution 10 times without issues.

      'parameterFull' and 'resultFull' implement an auto-format detection. It differentiates values between booleans, strings and numbers. Based on the log you provided, it appears that the result was originally identified as a number (consistent with settings above) but the new build result is identified as a string.

      My suggestion is to access the result file inside the workspace and check the nature of the result for the failed build.

      If the result contains a string, to continue using the benchmark with this job, you will need to delete the corrupted build.

      Let me know,

      Daniel M

  6. Ivan Malamen

    Hi Daniel!

    I'm trying to use benchmark pipeline step

    benchmark inputLocation: 'test/benchmark/output/*.xml', schemaSelection: 'default', truncateStrings: true

    , but fail with 

    Error: java.io.IOException: The selected schema does not exist.

    What are available values for schema? It is not listed in docs.

    I've tried "simplest","default","simplest.xml","defulat.xml" but no success...


    1. Daniel Mercier

      Hi Ivan,

      Sorry about that, pipeline implementation came late into the development and there is a mismatch between what the pipeline script expect and the displayed values in the standard job. The script equivalents are: 

      • 'Default' mode =  'defaultSchema' as schemaSelection in the pipeline script.
      • 'Simplest' mode = 'simplestSchema'
      • 'Custom' mode = 'customSchema'

      My suggestion would be to use the 'Pipeline Syntax' tool. It is very handy to determine what plugins are available for pipelines as well as their correct syntaxes.

      Best regards,

      Daniel

      1. Ivan Malamen

        Hi  Daniel!


        Thank you for answer!

        I've tried "Pipeline syntax", but there is no object for "benchmark"...


        Thank you,

        Ivan

        1. Daniel Mercier

          Hi Ivan,

          Which version of the plugin do you use ? Pipeline support was finalized in version 1.0.6.

          Let me know,

          Daniel

          1. Ivan Malamen

            I installed 1.0.6 from Jenkins plugins store.

            1. Daniel Mercier

              Strange, it appears on my Jenkins deployments (now 2.138.2) in the Pipeline Syntax between:

              'bat: Windows bat script' and 'build: Build a job' as 'benchmark: Benchmark results'.

              By the way, I would suggest to update to 1.0.7 as it fixes a bug (not related to our conversation) and improves usability.

              Daniel