View CA-APM on the plugin site for more information.
For ca-apm 1.x please refer to CA APM Plugin 1.x
CA APM Plugin 2.x and later
Work in progress..
Jenkins is a tool to automate continuous delivery pipelines. A most common use of Jenkins is to run tests against the new builds of an application. The CA APM Jenkins plug-in extension automates comparing the performance of builds. The plug-in provides abstraction to run the plug-in after any LoadGenerator, define multiple comparison strategies and define multiple output handlers.
- LoadGenerator :You can use Jmeter and CA Balzemeter as the load generators
- Comparison Strategies: CA Jenkins plugin supports the following comparison strategies:
- Mean Latency: Compares the average value of a metric of a transaction with its corresponding value of benchmark build.
- Static Threshold: Compares the value of the metric with the corresponding threshold value defined in the configuration.
- Output Handlers: Email HTML Report, JSON, and Chart Output, Histogram Chart Output
- Ability to fetch multiple metrics using regex as KPI for tracking
- Ability to configure multiple fail conditions
- Metric A less than or greater than a constant value OR
- Metric A in comparison to Metric B
- Ability to send notification via Email (if smtp server is configured)
- Ability to view KPI's per build or cross build dashboard in Jenkins
- Ability to decorate APM Team Center 10.1 nodes using custom attributes like build status, number and date
- Ability to just notify via email: If "justEmail" is checked the build will NOT be marked as fail even if the conditions are met but only a notification will be sent
- Ability Jump to Webview in context (for both metric and time range) from build or x-build dashboard
Jenkins version 2.60.1 or newer
CA APM Jenkins plug-in Packaging
The CA APM Jenkins plug-in package contains the following folder and files:
ca-apm-jenkins.hpi (HPI file)
- javadoc directory - Contains java documentation about comparison-strategy, output-handler, and available helper utilities
Install the Jenkins plug-in
This section describes the steps to install the plug-in. You can install the Jenkins plug-in from the Jenkin's Manage Plug-ins page.
Follow these steps:
- Select Jenkins, Manage Jenkins, Manage plugins.
The Manage Plug-in page opens.
- Select the Advanced tab.
The HTTP Proxy Configuration page opens.
Browse and upload the performance-comparator-jenkins.hpi file available in the Repo directory.
Wait for the installation to complete.
Note: Ensure that no jobs are running
- Restart the Jenkins when installation is complete.
Using the Jenkins Pipeline Syntax to Run the CA APM Jenkins plug-in
To use BlazeMeter in pipeline syntax you need to have BlazeMeter Plugin
Sample pipeline scripts with jmeter. Please refer the Configure LoadGenerator When Using JMeter
Follow these steps:
- Add the predefined stage in your existing pipeline job.
- Run the Jenkins job.
Important: The Jenkins plug-in and the properties files must be in the same machine.
Configure properties in performance-comparator.properties file.
Download the performance-comparator.properties file from http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ca-apm/ca-apm-dist/<version>/ca-apm-dist-<version>.zip
Please refer the Configure the CA APM Jenkins Plugin Properties
Default Comparison Strategies and Output Handler Available with the plug-in
The plug-in has the following built-in Comparison Strategies and Output Handlers.
List of Available Comparison-Strategies
By default the following strategies are executed:
If you do not want to execute the default strategy, remove the corresponding strategy name from
comparisonstrategies.list property value list.
For example, for meanLatencyStrategy, it can be any RegEx like *Frontend.*Health: Average Response Time \\(ms\\). Do not remove Health: Average Response Time \\(ms\\), else it looses its logical meaning.
By default, the following comparison-strategies are available with the plug-in:
Mean Latency Comparison Strategy
The strategy compares the average value of the metric between current build and benchmark build with threshold of five percentage. You can set a custom value for the <metricName>
.threshold property. The following code shows the sample configuration:
Static Threshold Strategy
The strategy compares your Concurrent Invocation metric of transactions of the current build with the given threshold value (2).
List of Available Output-Handlers
The plug-in provides the following default output handlers that are mapped to all the following default comparison-strategies in the outputhandlers property. You can remove any or all of the output handlers. The output-handlers should be declared with the below property as below:
Plain Text Email Report Output Handler
The handler generates a plain text email comparison report. The report contains the metrics for all the strategies that are mapped that are in the properties file.
JSON File Output Handler
The handler generates a JSON report of the average value of the metrics comparison between current build and benchmark build as a percentage.
Chart Output Handler
The handler generates a Chart Output of comparison of performance metrics between benchmark build and current build.
The output is stored in
<current_build>/workspace/chartOutput folder. The output html files are in the
chartOutput/output folder. The folder contains one file for each comparison-strategy. For Example, If one comparison strategy has six metric paths that are identified, then the output file contains six charts.
Histogram Output Handler
The handler generates build to build comparison in histogram style, with the average value of each metric for each build, default it render for 10 builds.
Comparison Strategy to Output-Handler Mapping
The mapping lets you receive the output from only those
comparison-strategies which are mapped to output-handler using the
outputHandlers property. The communication between a comparison-strategy and output-handler happens using the Data Structure T within
StrategyResult Entity.Map the comparison-strategy and output-handler using the same data format. If the mapping is incorrect, it is not processed by Output-Handler and plug-in raises a warning.
The default implementation of T is DefaultStrategyResult provided with the plug-in. All the default strategies and output-handlers communicate using the
Jenkins Attributes in APM Team Center
With the help of Jenkins plug-in, you can view the following Jenkins attributes as Custom attributes to each application vertex:
Extending the plugin
Extending the plug-in lets you add new sub commands to the existing plug-in to perform new tasks. Please refer the Extending the CA APM Jenkins Plugin
Troubleshoot the plug-in
This section discusses the error message that you see on your screen, symptom, and solution to resolve the error.
Comparator Plugin Execution Completed with failures
The jenkins job fails and detailed specific error is printed on the Jenkins build console.
Review the comparison-runner.log file that is generated in
Multiple Error Messages for similar Symptom
- APM authentication configuration file does not exist
- LoadGenerator Configuration file does not exist
- Strategies Configuration file does not exist, file path is <file_path>
- System Configuration file does not exist, file path is <file_path>
Message printed on Jenkins Console Output
Input Properties file defined in parameters does not exist, please check
Jenkins plug-in run fails because the specific configuration file does not exist.
Place the file in the specified location as printed in the error message and run the plug-in.
Comparison Strategy handler for <strategy_name> is not defined
There is no value against the property omparator which is the comparison strategy handler.
Provide the complete qualified class name of the comparison strategy handler.
No Agent Specifier(s) defined for <strategy_name>
Values are not specified against agentspecifier for a given comparison strategy.
Provide at least one agent-specifier regular expression, at least .*
No metric specifier defined for <strategy_name>
Value is not specified against metricspecifier for a given comparison strategy.
Provide a qualified correct regular expression for metric specifier using the following syntax:
Warning: No output handler(s) mapped to <strategy_name>
The given comparison strategy is unmapped to any output-handler, so its output is not used anywhere.
Either comment this comparison-strategy or assign a relevant output-handler to this comparison-strategy.
No output-handler(s) defined in the configuration, hence exiting Exiting means will stop running the plugin further
No output handlers are mentioned against the outputhandlers list.
Provide at least one output-handler, which is mapped to at least one comparison-strategy.
class is not present in the classpath
The specified class is not present inside a jar in the extensions directory that is defined in the
Check if the specified class is present in the configured extensions directory.
Error in executing comparison strategy <strategy_name>
Detailed reason is present in comparison-runner.log file present in workspace current build directory.
Check for detailed error and act appropriately.
Error in executing Output strategy <strategy_name>
Detailed reason is present in comparison-runner.log file present in the
Check for detailed error and act appropriately.
Version 2.0-beta-0 (January 25, 2019)
- Initial release for 2.x