Plugin Information |
---|
View Benchmark on the plugin site for more information. |
Benchmark plugin to compare boolean and numerical results over the history of builds.
About
This plug-in adds a post-build capability.
This plug-in purpose is to benchmark over all of a job builds, boolean and numerical results through tables and graphs.
This plug-in has also a mechanism of thresholds over numerical values to control a build success or failure.
This plug-in is also designed to load any type of JSON or XML result file format.
This plug-in works best in combination with an automated notification plug-in.
This plug-in works also as a result viewer for the 'jUnit Plug-in' and the 'xUnit Plug-in'.
This plug-in is accessible as a Jenkins pipeline component.
The documentation on how to use the plugin is located inside the Github repository.
For the plug-in one slide features, click HERE.
If you are interested in why this plug-in was created, please find the original proposal HERE.
Simple mode - Two steps use
The Benchmark plug-in in its simplest form works as a result viewer for the 'jUnit Plug-in' and the 'xUnit Plug-in'.
- Add the benchmark plugin after a xUnit or jUnit post build step in the job configuration,
- Run the build.
The icon to access the plug-in result viewer should appear on the job main window.
Standard mode - Use registered schema
Here are the links to the standard schema:
To use one, just select it in the dropdown menu of the plug-in configuration. The necessary schema format (JSON or XML) will be automatically detected.
Advanced mode - Create your own schema
Use the manual to build a custom schema: HERE
A custom schema can be either present in the workspace or added to a text field during the job configuration.
Changelog
1.0.11
Closed InputStreams that led to threads being kept alive during import operations (Thank you Artalus S.).
1.0.10
Fixed detected XML parsing security issue.
1.0.9
Fixed issue where processing failed if the number of cores detected is 1 (Thank you Nick Charsley).
1.0.8
Fixed issue where values of String parameters appeared as N/A.
1.0.7
Removed hyphens from the main tables to allow numerical sorting.
Removed text versions (true/false) of Boolean results from the raw table to allow numerical sorting.
Adjusted left the Group/Result/Unit names for readability and prevent the awkward display.
Fixed a bug that occurred when changing schema content where old parameters ended as results and crashed the display of the main tables.
If the table display is frozen due to this bug, the workaround is:
- Update the plugin,
- Go where the job is stored on the master node [requires access rights],
- And, delete the file 'BenchmarkCondensed.json' in the job folder.
The file 'BenchmarkCondensed.json' will be recomposed during the next build.
Warning: Any result from past builds, absent in the new result file(s) will not be considered.
1.0.6
Following fixes by magik6k (Thank you).
Fixed issue with pipeline support.
Improved failure mechanism.
Added link inside README to a page on how to write the plugin script for pipeline.
1.0.5
Issues for last 3 updates identified by Shay Gal-on (Thank you)
Fixed issue with additional thresholds.
Fixed issue with result display.
Updated help text.
1.0.4
Fixed issue with multi-file processing.
Fixed issue with log content.
1.0.3
Fixed issue with ResultFull and ParameterFull for JSON.
Fixed issue with merge system.
Fixed issue with log content.
1.0.2
Switched loading of schema file from File to String; to InputStream to String.
Fixed issue preventing the link between condensed table and individual result pages.
1.0.1
First official release.
0.8.4
Initial commit as OpenSource under MIT license.
Added pipeline support.
Added Jenkinsfile for the plugin to be built by the Jenkins CI Infrastructure and check pull requests.
Activated SCM.
Fixed Findbugs warnings.
Added Findbugs exclude filter for specific warnings (see /src/findbugs/excludesFilter.xml for details).
30 Comments
Unknown User (shaygalon)
Is it possible to provide an example output file for the json?
The simplest file that an application can create that will result in the plugin parsing it and providing output.
e.g. how to convert something along the lines of:
my_benchmark_metric_1,57
my_benchmark_metric_2,32
Into something for the plugin to consume.
In fact, if it could consume a simple csv that would be great!
Unknown User (damercie)
Hi Shay,
For the result file:
the schema should be:
For all-in-one "key":"value", there are two JSON specific types called resultFull and parameterFull.
Cheers,
Daniel
Unknown User (shaygalon)
Looks simple enough, but when selecting "custom" for schema, there is no way to feed in that custom schema.
What am I missing?
Unknown User (shaygalon)
NVM. Found it
Unknown User (shaygalon)
Still not working
Created a simple job that does that:
echo "{ 'throughput' : 10, 'latency' : 0 }" > tracking.json
Pick it up on postbuild with custom schema:
{
"description": "Performance tracking",
"failure": { "value": true },
"type": "object",
"properties": {
"throughput": {"type": "resultFull"},
"latency": {"type": "resultFull"}
}
}
And when run I see:
Unknown User (damercie)
Your JSON content is incorrect. Use instead:
Unknown User (shaygalon)
Same thing.
From the workspace in the machine:
$ cat tracking.json
{ "throughput" : 2, "latency" : 0 }
Same output from the benchmark plugin
Unknown User (damercie)
Hi Shay,
You found a bug. I identified two more during the fixing. It should work with version 1.0.3. The version was pushed to Jenkins Artifactory but it will take some time to become available on Jenkins. It will appear as an update.
Thank you for finding the bug.
Daniel
Unknown User (shaygalon)
Users are the worst thing that can happen to software aren't they...
Unknown User (damercie)
Sorry mate,
I have to disagree. I had to expedite originally this project because of another one; and so, I am glad that you took the time and had the patience to help me better the plugin.
I am not sure I will be able to introduce support for two column CSV files anytime soon but I like the idea. If you get the chance, please log the improvement on Github.
Let me know if you face any other issue or have any question.
Cheers,
Daniel
Unknown User (shaygalon)
gladly, though you will have to enable issues on your github project if you want anyone to log an issue...
Unknown User (damercie)
Fair point. It is active now.
Thanks,
Daniel
Unknown User (shaygalon)
How do I get "groups"? Not clear from the docs what needs to be added.
Thanks, Shay
Unknown User (damercie)
Hi Shay,
A 'group' is simply a nesting object or nesting array declared as type 'object' or 'array' in the schema. It follows/records the JSON tree. There can be any depths of sub-groups. A group is named using the nesting JSON key but the name can also be customized using a nested key:value declared in the schema as 'type:name'.
Note: There can be an issue with the display of results if the depth (number of subgroups) is very high or the group keys/names are very long (will also depend on the screen size).
Regards,
Daniel
Unknown User (damercie)
Here is a modified version of your example (Windows batch):
Inside the benchmark plugin configurations:
Result file:
Custom schema:
You should now get a group named 'test' in the result tables.
Regards,
Daniel
Unknown User (shaygalon)
Is it possible to make the group name non static?
Use case is to separate tests that run under different conditions. Think parametrized build, you want to group results with specific parameters together as they are not directly comparable...
Unknown User (damercie)
Hi Shay,
Do you have an example in mind ?
For separate groups of tests with different conditions in the same result file, I would use something like:
with schema:
Note: Parameters can be grouped and still be attached to the result(s) next to the parameter group.
Daniel
Unknown User (shaygalon)
I would like something where the test run can have:
{
"$PARAM"
: {
"time"
:
"5"
,
"L"
:
12
}
}
PARAM is a build parameter, and I don't want to have a different schema for every possible value of that parameter.
In the simplest case, think of grouping by the node the test is executing on, or by the compiler version used, or dataset used. You don't want a new job every time you make a change, and similarly you don't want to edit your schema. Is it possible to do something along the lines of:
{
"group_by"
: {
"group_name"
:
"$PARAM"
,
"L"
:
12
}
}
Unknown User (damercie)
Hi Shay,
Sounds interesting, very dynamic and especially useful for filtering purpose.
On the down side, results will become disconnected from one build to another because group path changed. File paths and group paths are used to identify and connect results between builds. This would also seriously affect the advantage of the result page (click a result line on one of the tables) by preventing the correlation between parameter changes and result changes.
If this feature is really important to you, it can still be roughly implemented as described in your message with the current version of the plugin using one of the two approaches:
Honestly, I don't think we should implement it like this. However, I can see an equivalent that could fit (let me know). It is something I considered before; but decided not to implement because of the overhead and because I was pressed by time.
The idea was to leverage 'array' to create N-dimension results. It would allow the display of 'result vs parameter vs build' instead of the current 'result vs build'. At the moment, an 'array' is considered as the holder of multiple results with similar format and uses the index as group name to identify/differentiate results. This behavior could be changed to hold the N dimensions of a result.
I will look into it but I have no ETA.
I thank you in advance for your understanding and continued interest in the plugin,
Daniel
Unknown User (shaygalon)
Actually tried the parameter as filename, but the current layout makes that quite unwieldy
also the benefit of grouping is not there.
Currently using exactly a schema that encodes lots of parameter values, but it seems counter intuitive and likely to end up in missing results because of using a different parameter value.
I like the idea of using arrays to represent this, and you can then filter on specific values on the array as well, which would make perfect sense. Understood that this would be a more significant effort, was really trying to figure out if this can be achieved by some smart schema but seems like a limitation of current implementation.
Thanks again for your responsiveness!
Unknown User (shaygalon)
what does parameterFull do, and where does it show up in results
Unknown User (damercie)
If you recall above:
Two results called "L" will appear under group_1 and group_2.
To visualize the parameters, access a result page by clicking on a result line inside one of the tables.
Regards,
Daniel
Unknown User (rgadgil)
I have schema like
{
"description": "Performance tracking",
"failure": {"value": true},
"type": "object",
"properties": {
"G1": {
"type":"object",
"properties":{
"T": {"type": "parameterFull"},
"g": {"type": "resultFull"}
}
},
"G2": {
"type":"object",
"properties":{
"T": {"type": "parameterFull"},
"g": {"type": "resultFull"}
}
}
}
}
my json results file is:
{"G1":{"T" : 180, "g" : 946}}
I get the following error
Any insights on this?
Unknown User (damercie)
Hi Rajeev,
I used the information you provided and ran the execution 10 times without issues.
'parameterFull' and 'resultFull' implement an auto-format detection. It differentiates values between booleans, strings and numbers. Based on the log you provided, it appears that the result was originally identified as a number (consistent with settings above) but the new build result is identified as a string.
My suggestion is to access the result file inside the workspace and check the nature of the result for the failed build.
If the result contains a string, to continue using the benchmark with this job, you will need to delete the corrupted build.
Let me know,
Daniel M
Unknown User (ivanmalamen)
Hi Daniel!
I'm trying to use benchmark pipeline step
benchmark inputLocation: 'test/benchmark/output/*.xml', schemaSelection: 'default', truncateStrings: true
, but fail with
What are available values for schema? It is not listed in docs.
I've tried "simplest","default","simplest.xml","defulat.xml" but no success...
Unknown User (damercie)
Hi Ivan,
Sorry about that, pipeline implementation came late into the development and there is a mismatch between what the pipeline script expect and the displayed values in the standard job. The script equivalents are:
My suggestion would be to use the 'Pipeline Syntax' tool. It is very handy to determine what plugins are available for pipelines as well as their correct syntaxes.
Best regards,
Daniel
Unknown User (ivanmalamen)
Hi Daniel!
Thank you for answer!
I've tried "Pipeline syntax", but there is no object for "benchmark"...
Thank you,
Ivan
Unknown User (damercie)
Hi Ivan,
Which version of the plugin do you use ? Pipeline support was finalized in version 1.0.6.
Let me know,
Daniel
Unknown User (ivanmalamen)
I installed 1.0.6 from Jenkins plugins store.
Unknown User (damercie)
Strange, it appears on my Jenkins deployments (now 2.138.2) in the Pipeline Syntax between:
'bat: Windows bat script' and 'build: Build a job' as 'benchmark: Benchmark results'.
By the way, I would suggest to update to 1.0.7 as it fixes a bug (not related to our conversation) and improves usability.
Daniel