DX APM Jenkins Plugin

CA APM Plugin 2.x and later

Jenkins is a tool to automate continuous delivery pipelines. A most common use of Jenkins is to run tests against the new builds of an application. The CA APM Jenkins plug-in extension automates comparing the performance of builds. The plug-in provides abstraction to run the plug-in after any LoadGenerator, define multiple comparison strategies and define multiple output handlers.

• LoadGenerator :You can use any load generators example - Jmeter and CA Balzemeter
• Comparison Strategies: CA Jenkins plugin supports the following comparison strategies:
  • Mean Latency: Compares the average value of a metric of a transaction with its corresponding value of benchmark build.
  • Static Threshold: Compares the value of the metric with the corresponding threshold value defined in the configuration.
• Output Handlers: Email HTML Report, JSON, and Chart Output, Histogram Chart Output

Features

• Ability to fetch multiple metrics using regex as KPI for tracking
• Ability to configure multiple fail conditions
  • Metric A less than or greater than a constant value OR
  • Metric A in comparison to Metric B
• Ability to send notification via Email (if smtp server is configured)
• Ability to view KPI's per build or cross build dashboard in Jenkins&nbsp
• Ability to decorate APM Team Center 10.1 nodes using custom attributes like build status, number and date
• Ability to just notify via email: If "build.fail" property is false the build will NOT be marked as fail even if the conditions are met but only a notification will be sent
• Ability Jump to Webview in context (for both metric and time range) from build or x-build dashboard

Requirements

Jenkins

Jenkins version 2.60.1 or newer

CA APM Jenkins plug-in Packaging

The CA APM Jenkins plug-in package contains the following folder and files:

• Properties
  • performance-comparator.properties
• Repo
  • ca-apm-jenkins.hpi (HPI file)
  • ca-apm-api.jar
  • ca-apm-core.jar
  • ca-apm-default-strategy.jar
• javadoc directory - Contains java documentation about comparison-strategy, output-handler, and available helper utilities

CA APM Agent Setup

Agent need to be configured for the Application for which user running the JMeter and Blazemeter  . For More info on Configure and Download an Agent

You must configure the application server that you are instrumenting to include the path to the agent primary .jar file (Agent.jar)  Usually, you configure the path by editing the application server startup script, then restarting the application server. When the application server restarts, the Java Agent instruments the classes discovered for default components of the JVM and application environment. The specific steps that are involved depend on the application server.  find additional info Configure Application Server to Use the Java Agent.

Install the Jenkins plug-in

This section describes the steps to install the plug-in. You can install the Jenkins plug-in from the Jenkin's Manage Plug-ins page.

Follow these steps:

1. Select Jenkins, Manage Jenkins, Manage plugins.
   
   The Manage Plug-in page opens.
2. Select the Advanced tab.
   
   The HTTP Proxy Configuration page opens.

3. Browse and upload the performance-comparator-jenkins.hpi file available in the Repo directory.
   
   Wait for the installation to complete.

   Note: Ensure that no jobs are running

4. Restart the Jenkins when installation is complete.

Using the Jenkins Pipeline Syntax to Run the CA APM Jenkins plug-in

  Jenkins job can be configured with any load generator to generate the load on the application, as below, before running the ca-apm plugin. 

  Example : 

Blazemeter

To use BlazeMeter in pipeline syntax you need to have BlazeMeter Plugin

Please refer the below sample pipeline Blazemeter script from APM 10.x.

JMeter

Sample pipeline scripts with JMeter for APM 10.x

pipeline {   
   agent any     
   stages {
	stage('JMeter'){ 
		steps{   
            script {
                 loadGeneratorName = env.STAGE_NAME;
                 loadGeneratorStartTime = System.currentTimeMillis();
                 sh "${env.WORKSPACE}\\<jmeter_home>\\bin\\jmeter.sh -n -t <jmx_file_path>"
                 loadGeneratorEndTime = System.currentTimeMillis();
                map = [jenkinsPluginName: "CAAPM"];
            }              
        }
    }            
   stage('CAAPMPerformanceComparator') {
      steps {  
           caapmplugin performanceComparatorProperties:"${env.WORKSPACE}/properties/performance-comparator.properties",
           loadGeneratorStartTime: "$loadGeneratorStartTime",
           loadGeneratorEndTime: "$loadGeneratorEndTime",
           loadGeneratorName: "$loadGeneratorName",
           attribsStr: "$map"; 

     }
   }   
  }
}

Follow these steps:

1. Add the predefined stage in your existing pipeline job.
2. Run the Jenkins job.

Important: The Jenkins plug-in and the properties files must be in the same machine.

Plugin Configuration

Configure properties in performance-comparator.properties file.

Download the performance-comparator.properties file from  http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ca-apm/ca-apm-dist/<version>/ca-apm-dist-<version>.zip

Please refer the Configure the CA APM Jenkins Plugin Properties

Default Comparison Strategies and Output Handler Available with the plug-in

The plug-in has the following built-in Comparison Strategies and Output Handlers.

List of Available Comparison-Strategies

By default the following strategies are executed:

• MeanLatencyStrategy 
• staticThresholdStrategy

If you do not want to execute the default strategy, remove the corresponding strategy name from comparisonstrategies.list property value list.

For example, for meanLatencyStrategy, it can be any RegEx like *Frontend.*Health: Average Response Time \\(ms\\). Do not remove Health: Average Response Time \\(ms\\), else it looses its logical meaning.

By default, the following comparison-strategies are available with the plug-in:

Mean Latency Comparison Strategy

The strategy compares the average value of the metric between current build and benchmark build with threshold of five percentage. You can set a custom value for the <metricName>.threshold property. The following code shows the sample configuration:

#MeanLatency Comparison-Strategy
#threshold value, e.g.,2. the build will fail,if the build.fail property value is set true and if the metric's avg value difference
#between current and benchmark build cross this value
cpu.threshold=5
#agent name of the application,  e.g., .* means any agent and multiple agent specifiers can be given with comma separation
cpu.agentspecifier=.*
#metric path , it will be specific to agent, application. e.g, .*CPU.*Processor 0:Utilization % \\(aggregate\\), multiple metric specifiers can be given with comma separation
cpu.metricspecifier=.*CPU.*Processor 0:Utilization % \\(aggregate\\)
#comparator class name excluding the "ComparisonStrategy", e.g., MeanLatency for MeanLatencyComparisonStrategy
cpu.comparator=MeanLatency
#list of output handlers for this strategy. available output handlers : plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml
cpu.outputhandlers=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml

Static Threshold Strategy

The strategy compares your Concurrent Invocation metric of transactions of the current build with the given threshold value (2).

#StaticThreshold Comparison-Strategy
#threshold value,e.g., 2.  it will be compared with the average value of metric for current  build
concurrentinvocations.threshold=2
#agent name of the application, e.g., .*  and multiple agent specifiers can be given with comma separation
concurrentinvocations.agentspecifier=.*
#metric path , it will be specific to agent, application. e.g., .*Business Segment.*Health:Concurrent Invocations. multiple metric specifiers can be given with comma separation
concurrentinvocations.metricspecifier=.*Business Segment.*Health:Concurrent Invocations
#comparator class name excluding the "ComparisonStrategy", e.g., StaticThreshold for StaticThresholdComparisonStrategy
concurrentinvocations.comparator=StaticThreshold
#list of output handlers for this strategy. available output handlers : plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml
concurrentinvocations.outputhandlers=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml

List of Available Output-Handlers

The plug-in provides the following default output handlers that are mapped to all the following default comparison-strategies in the outputhandlers property. You can remove any or all of the output handlers. The output-handlers should be declared with the below property as below:

outputhandlers.list=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml

Plain Text Email Report Output Handler

The handler generates a plain text email comparison report. The report contains the metrics for all the  strategies that are mapped that are in the properties file.

JSON File Output Handler

The handler generates a JSON report of the average value of the metrics comparison between current build and benchmark build as a percentage.

Chart Output Handler

The handler generates a Chart Output of comparison of performance metrics between benchmark build and current build.

The output is stored in <current_build>/workspace/chartOutput folder. The output html files are in the chartOutput/output folder. The folder contains one file for each comparison-strategy. For Example, If one comparison strategy has six metric paths that are identified, then the output file contains six charts.

Histogram Output Handler

The handler generates build to build comparison in histogram style, with the average value of each metric for each build, default it render for 10 builds.

Comparison Strategy to Output-Handler Mapping

The mapping lets you receive the output from only those comparison-strategies which are mapped to output-handler using the outputHandlers property. The communication between a comparison-strategy and output-handler happens using the Data Structure T within StrategyResult Entity.Map the comparison-strategy and output-handler using the same data format. If the mapping is incorrect, it is not processed by Output-Handler and plug-in raises a warning.

The default implementation of T is DefaultStrategyResult provided with the plug-in. All the default strategies and output-handlers communicate using the StrategyResult<DefaultStrategyResult> Entity.

DX APM SaaS Integration

Jenkins Attributes in APM Team Center

With the help of Jenkins plug-in, you can view the following Jenkins attributes as Custom attributes to each application vertex:

• currentBuildNumber
• benchmarkBuildNumber
• buildStatus
• loadGeneratorStartTime
• loadGeneratorEndTime

Extending the plugin 

Extending the plug-in lets you add new sub commands to the existing plug-in to perform new tasks. Please refer the Extending the CA APM Jenkins Plugin 

Troubleshoot the plug-in

This section discusses the error message that you see on your screen, symptom, and solution to resolve the error.

Comparator Plugin Execution Completed with failures

Symptom:

The jenkins job fails and detailed specific error is printed on the Jenkins build console.

Solution:

Review the comparison-runner.log file that is generated in <current_build>/workspace directory.

Message printed on Jenkins Console Output

Input Properties file defined in parameters does not exist, please check

Symptom:

Jenkins plug-in run fails because the specific configuration file does not exist.

Solution:

Place the file in the specified location as printed in the error message and run the plug-in.

Comparison Strategy handler for <strategy_name> is not defined

Symptom:

There is no value against the property comparator which is the comparison strategy handler.

Solution:

Provide the complete qualified class name of the comparison strategy handler.

No Agent Specifier(s) defined for <strategy_name>

Symptom:

Values are not specified against agentspecifier for a given comparison strategy.

Solution:

Provide at least one agent-specifier regular expression, at least .*

No metric specifier defined for <strategy_name>

Symptom:

Value is not specified against metricspecifier for a given comparison strategy.

Solution:

Provide a qualified correct regular expression for metric specifier using the following syntax:

Business Segment\|<appName>\|[^|]+\|Health:Average Response Time \(ms\)

Warning: No output handler(s) mapped to <strategy_name>

Symptom:

The given comparison strategy is unmapped to any output-handler, so its output is not used anywhere.

Solution:

Either comment this comparison-strategy or assign a relevant output-handler to this comparison-strategy.

No output-handler(s) defined in the configuration, hence exiting Exiting means will stop running the plugin further

Symptom:

No output handlers are mentioned against the outputhandlers list.

Solution:

Provide at least one output-handler, which is mapped to at least one comparison-strategy.

class is not present in the classpath

Symptom:

The specified class is not present inside a jar in the extensions directory that is defined in the performance-comparator.properties file.

Solution:

Check if the specified class is present in the configured extensions directory.

Error in executing comparison strategy <strategy_name>

Symptom:

Detailed reason is present in comparison-runner.log file present in workspace current build directory.

Solution:

Check for detailed error and act appropriately.

Error in executing Output strategy <strategy_name>

Symptom:

Detailed reason is present in comparison-runner.log file present in the <current_build>/workspace directory.

Solution:

Check for detailed error and act appropriately.

Changelog

Release 2.1.2 (September 5, 2019) 

• Addressed the issue for DX APM Links from output handlers
• Addressed Custom Attributes in ATC

Release 2.1.1 (August 23, 2019) 

• Added Support for DX APM 11.1 and DX SaaS
• Code Changes to support for API changes in latest version of DX APM 11.1 and DX SaaS
• Code changes to support sending events to OI
• Publish SCM(source code management) parameters as Attributes
• Compatibility with APM 10.x releases
• Sev 1 issue of release 2.1 is addressed

Known Issues

• DXAPM URL link broken 
• Update of Custom Attributes in ATC

Release 2.1 (August 22, 2019) 

Known Issues: 

• Sev1 : Error while fetching BuildPerformanceData

Note: Please use release 2.1.2 for APM 10.x , DX APM 11.1 and DX SaaS

Release 2.0 (May 27, 2019)

• 2.0 release with reading load generator time from Jenkins job phase

Release 2.0-beta-0 (January 25, 2019)

• Initial release for 2.x