Jenkins : TraceTronic ECU-TEST Plugin

Table of contents

Features

• Provides an easy integration and control of ECU-TEST and the Tool-Server with Jenkins
• Enables the execution of ECU-TEST packages and projects with their respective configurations

Moreover publishing test results is included in form of:

• Linking the TRF reports
• Generating various report formats
• Displaying test result trend graphs
• Uploading ATX test reports to TEST-GUIDE
• Parsing the ECU-TEST logs for warnings and errors
• Downstream report generation

Configuration

ECU-TEST configuration

• ECU-TEST installations are administrated in the Jenkins system configuration or starting with Jenkins 2.0 in the global tool configuration at section "ECU-TEST"
• An installation entry is specified by an arbitrary name and the path to the installation directory
• The execution on a Jenkins slave requires the adaption of the ECU-TEST installation directory on the slave configuration page

TEST-GUIDE configuration

• TEST-GUIDE is also configured in the Jenkins system configuration at section "TEST-GUIDE"
• A new entry requires both an arbitrary name and the selection of a previously configured ECU-TEST installation
• Further settings like the server or ATX specific configuration can be found in the advanced block

Job configuration

The plugin provides several new build steps:

Note:

• All build steps have an "Advanced" button through which more options can be accessed
• All text fields can be parameterized and are resolved at build execution
  Syntax: $parameter or ${parameter}

The post-build actions are extended by these test report related post-build steps:

Job execution

• After job configuration is done a new build can be triggerd by clicking on the "Build now" button
• The console log contains detailed information per build step execution
• The test results are published both on the build page and the job page based on the latest build

Environment variables

• This plugin also exposes the tool and test related information of the current build as environment variables
• These variables are composed of $TT_<TOOL | TEST>_<information>_n where n is the 0-based index of the related build steps in the job configuration
• The summary of all generated environment variables is also shown on the build page
  
  

Downstream report generation

The concept of downstream report generation allows to execute the report generators like ATX, UNIT, HTML etc. in a downstream-based project.
Therefore the report artifacts like TRF reports etc. must be archived after the test execution stage and will then be copied into a separate workspace of the triggered downstream project.

This downstream project acts like a central reporting service which waits for inputs from other test executions. It will be an ease to scale this service with help of Jenkins parallelization and distribution features.
The big advantage of this approach is to save time for the report generation and uploading process on expensive testing resources like HiL test benches. These resources can be directly used for the next test.

Example configurations

Preconditions

To prevent duplicate reports the build workspace should be cleaned before running the tests and before copying artifacts inside of the downstream project.
This can be achieved by using clean checkouts from repository or using the Workspace Cleanup Plugin when preparing the build environment.

To transfer the report artifacts from one node to another the Copy Artifact Plugin is required for freestyle projects. When using pipelines also stash/unstash commands can be used.

Freestyle configuration

Executing tests and archiving the artifacts (upstream)

Copying artifacts and generating ATX report (downstream)

Pipeline configuration

node('windows') {
    deleteDir()
    
    startET 'ECU-TEST'
    testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
    stopET 'ECU-TEST'
    
    archiveArtifacts 'TestReports/**/*.trf'
    build job: 'downstream', wait: false
}

node('windows') {
    deleteDir()
    
    copyArtifacts filter: 'TestReports/**/*.trf', projectName: 'upstream', selector: lastSuccessful()
    
    // Class notation is required for publisher steps inside downstream wrapper!
    // Available: ATXPublisher, ETLogPublisher, JUnitPublisher, ReportGeneratorPublisher, TMSPublisher, TRFPublisher
    downstreamPublisher workspace: '', publishers: [
        [$class: 'ATXPublisher', atxName: 'ECU-TEST'],
        [$class: 'JUnitPublisher', toolName: 'ECU-TEST']]
}

Downstream trace analysis

In addition to downstream report generation the plugin allows to run trace analyses in a downstream-based project. Therefore the test cases are executed without trace analysis at first.
Afterwards all trace analysis artifacts like analysis job files, traces, trace step templates and TRF reports etc. must be archived and will then be copied into a separate workspace of the triggered downstream project.

This downstream project acts like a central trace analysis service which waits for inputs from other test executions and is designed to be highly scalable.
The huge advantage of better ressource efficiency is the same as for downstream report generation and can also be used in combination.

Example configurations

Preconditions

Same as in downstream report generation.

Freestyle configuration

Executing test cases without trace analysis and archiving the artifacts (upstream)

Copying artifacts and running the trace analysis (downstream)

Pipeline configuration

node('windows') {
    deleteDir()
    
    startET 'ECU-TEST'
    testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf'], packageConfig: [runTraceAnalysis: false]
    stopET 'ECU-TEST'
    
    archiveArtifacts 'TestReports/**/*.trf, TestReports/**/*.ajob, Packages/**, Traces/**, TraceStepTemplates/**'
    build job: 'downstream', wait: false
}

node('windows') {
    deleteDir()
    
    copyArtifacts filter: 'TestReports/**/*.trf, TestReports/**/*.ajob, Packages/**, Traces/**, TraceStepTemplates/**', projectName: 'upstream', selector: lastSuccessful()
    
    // Class notation is required for publisher steps inside downstream wrapper!
    downstreamPublisher workspace: '', publishers: [
        [$class: 'TraceAnalysisPublisher', toolName: 'ECU-TEST', mergeReports: true]]
}

Extensions

Job DSL

The Job DSL Plugin allows the programmatic creation of projects using a DSL. This plugin provides some specific DSL extensions.

Furthermore, if there is a method which is currently not supported then Job DSL also provides an auto-generator.

steps {
  // Start ECU-TEST
  startET(String installation) {
    workspaceDir(String directory)
    settingsDir(String directory)
    timeout(int | String timeout = 120)
    debugMode(boolean debugMode = false)
    keepInstance(boolean keepInstance = false)
 }
  // Stop ECU-TEST
  stopET(String installation) {
    timeout(int | String timeout = 30)
  }
  // Start Tool-Server
  startTS(String installation) {
    toolLibsIni(String toolLibsPath)
    tcpPort(int | String port = 5017)
    timeout(int | String timeout = 120)
    keepInstance(boolean keepInstance = false)
 }
  // Stop Tool-Server
  stopTS(String installation)
    timeout(int | String timeout = 30)
  }
}

steps {
  // Run package
  testPackage(String packageFile) {
    testConfig {
      tbcFile(String tbcFile)
      tcfFile(String tcfFile)
      forceReload(boolean forceReload = false)
      loadOnly(boolean loadOnly = false)
      constants {
        constant(String name, String value)
        constant {
          name(String name)
          value(String value)
        }
      }
    }
    packageConfig {
      runTest(boolean runTest = true)
      runTraceAnalysis(boolean runTraceAnalysis = true)
      parameters {
        parameter(String name, String value)
        parameter {
          name(String name)
          value(String value)
        }
      }
    }
    executionConfig {
      timeout(int | String timeout = 3600)
      checkTestFile(boolean checkTestFile = true)
      stopOnError(boolean stopOnError = true)
    }
  }
  // Run project
  testProject(String projectFile) {
    testConfig { (...) }
    projectConfig {
      execInCurrentPkgDir(boolean currentPkgDir = false)
      filterExpression(String expression)
      // modes: NO_EXECUTION(0), SEQUENTIAL_EXECUTION(1), 
      // PARALLEL_EXECUTION(2), SEPARATE_SEQUENTIAL_EXECUTION(5), 
      // SEPARATE_PARALLEL_EXECUTION(6), NO_TESTCASE_EXECUTION(9)
      jobExecutionMode(int | String execMode = 1)
    }
    executionConfig { (...) }
  }
  // Run test folder
  testFolder(String testFolder) {
    // modes: PACKAGES_ONLY, PROJECTS_ONLY, PACKAGES_AND_PROJECTS
    scanMode(String mode = 'PACKAGES_AND_PROJECTS')
    recursiveScan(boolean recursive = false)
    testConfig { (...) }
    packageConfig { (...) }
    projectConfig { (...) }
    executionConfig { (...) }
  }
} 

steps {
  // Import projects
  importProjects {
    importFromArchive(String archivePath, String importPath, String importConfigPath, boolean replaceFiles)
    importFromArchive(String archivePath) {
      importPath(String importPath)
      importConfigPath(String importConfigPath)
      replaceFiles(boolean replaceFiles)
    }
    importFromTMS(String credentialsId, String projectPath, String importPath, int | String timeout = 60)
    importFromTMS(String credentialsId, String projectPath) {
      importPath(String importPath)
      timeout(int | String timeout = 60)
    }
    importFromTMSDir(String credentialsId, String projectDirPath, String importPath, int | String timeout = 60)
    importFromTMSDir(String credentialsId, String projectDirPath) {
      importPath(String importPath)
      timeout(int | String timeout = 60)
    }
    importAttributesFromTMS(String credentialsId, String projectPath, int | String timeout = 60)
    importAttributesFromTMS(String credentialsId, String projectPath) {
      timeout(int | String timeout = 60)
    }
  }  
  // Import packages
  importPackages {
    importFromTMS { (...) }
    importFromTMSDir { (...) }
    importAttributesFromTMS { (...) }
  }
  // Export projects
  exportProjects {
    exportToTMS(String credentialsId, String projectPath, String exportPath, boolean createNewPath, int | String timeout = 60)
    exportToTMS(String credentialsId, String projectPath) {
      exportPath(String exportPath)
      createNewPath(boolean createNewPath)
      timeout(int | String timeout = 60)
    }
    exportAttributesToTMS(String credentialsId, String projectPath, int | String timeout = 60)
    exportAttributesToTMS(String credentialsId, String projectPath) {
      timeout(int | String timeout = 60)
    }
  }  
  // Export packages
  exportPackages {
    exportToTMS { (...) }
    exportAttributesToTMS { (...) }
  }
}

publishers {
  // Publish ATX reports
  publishATX(String atxInstallation) {
    // Archive options
    allowMissing(boolean allowMissing = false)
    runOnFailed(boolean runOnFailed = false)
    archiving(boolean archiving = true)
    keepAll(boolean keepAll = true)
  }
  // Publish TRF reports
  publishTRF() {
    // Archive options...
  }
  // Publish UNIT reports
  publishUNIT(String installation) {
    unstableThreshold(double threshold = 0)
    failedThreshold(double threshold = 0)
    // Archive options...
  }
  // Publish ECU-TEST logs
  publishETLogs() {
    unstableOnWarning(boolean markUnstable = false)
    failedOnError(boolean markFailed = false)
    testSpecific(boolean testSpecific = false)
    // Archive options...
  }
  // Publish Report Generators
  publishGenerators(String installation) {
    generators {
      generator(String generatorName) {
        settings {
          setting(String name, String value)
          setting { 
            name(String name)
            value(String value)
          }
        }
      }
    }
    customGenerators { 
       customGenerator(String generatorName) { (...) }
    }
    // Archive options...
  }
  // Publish to Test Management System
  publishTMS(String installation, String credentialsId) {
    timeout(int | String timeout = 60)
    // Archive options...
  }
}

Pipeline

The Pipeline Plugin allows to orchestrate automation, simple or complex, using a DSL. This plugin supports the use of all provided build steps and post-build actions from within a Jenkins Pipeline build. The appropriate DSL syntax for these steps and actions can be easily generated with help of the Pipeline Snippet Generator using the related custom pipeline step or the 'step' command, when using plugin version prior to 1.13.

node('windows') {
    stage('Start Tools') {
        startET 'ECU-TEST'
    }
    stage('Test Execution') {
        testPackage testFile: 'sample.pkg', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
        testProject testFile: 'sample.prj', testConfig: [tbcFile: 'sample.tbc', tcfFile: 'sample.tcf']
    }
    stage('Publish Reports') {
        publishATX 'TEST-GUIDE'
    }
    stage('Stop Tools') {
        stopET 'ECU-TEST'
    }
}

node('windows') {
	// ECU-TEST installation using global variable ET
    def et = ET.newInstallation toolName: 'ECU-TEST', installPath: 'C:\\Program Files\\ECU-TEST 8.0'

	// or getting existing installation from global tool configuration
	def et = ET.installation('ECU-TEST')

	// Start ECU-TEST
	et.start('C:\\Data\\workspace', 'C:\\Data\\settings')

    // Stop ECU-TEST
    et.stop()
}

node('windows') {
    // Start tools, execute tests
	...
	
	// TEST-GUIDE server instantiation using global variable ATX
    def atx = ATX.newServer atxName: 'TEST-GUIDE', toolName: 'ECU-TEST', serverUrl: 'http://localhost:8085', uploadToServer: false, authKey: 'xxx', projectId: '1'

	// or getting existing instance from global configuration
	def atx = ATX.server('TEST-GUIDE')

	// Publish ATX reports directly
	atx.publish()
}

Screencasts

For more information about installation and configuration of the plugin as well as using the plugin in a slave-based setup see our screencasts.

Issues

To report a bug or request an enhancement to this plugin please create a ticket in JIRA.

Known limitations

When using the plugin in a slave-based setup (especially running the slave at command line via JNLP) you also need to restart the slave when restarting the master.
This is due to already loaded libraries, for more information see this blog post and related issue JENKINS-31961.

Release 1.8 implemented an extension point that will workaround this issue. In order to work a new task has to be created in the Windows Task Scheduler named RESTART_JENKINS_SLAVE and configured with actions how to restart the slave.

FAQ

When encountering problems or error messages, please check the installed plugin version at first and update to the most recent version, if any.
If the problem still exists search the following list of issues for possible solutions, otherwise you are asked to create a ticket in JIRA.

Compatibility

• Jenkins LTS 2.60.3 or higher
• Java SE Runtime Environment 8 or higher
• ECU-TEST 7.0.0 or higher
• optional: TEST-GUIDE 1.55.0 or higher

License

This plugin is licensed under 3-clause BSD license.
More information can be found inside the LICENSE file.

Changelog

Release 2.10 (Jul 03, 2019)

 Dynamic TEST-GUIDE installation not found when publishing reports (PR #69)

Rename pipeline symbols to camel case names (PR #68)

Dependencies updates (PR #63, #66)

Release 2.9 (Jun 14, 2019)

 Add generator option to read from persisted settings file (PR #44)
 Add support for analysis packages (PR #45)

Dynamic ECU-TEST installation not found when stopping instance (PR #43)

 Enable Dependabot (PR #49)
Enable Release Drafter (PR #52)

Release 2.8 (May 24, 2019)

 Check whether configurations are started (PR #32)
 Add timeout option using Quit and Exit via COM API (PR #34)
 Add more debug logs (PR #37)

Ensure compatibility up to ECU-TEST 8.0.0 (PR #35)
Drop support for ECU-TEST 6.x (PR #35)
Update available ATX settings to TEST-GUIDE 1.67.0 (PR #36)

Release 2.7 (May 17, 2019)

 Add full support for Configuration as Code plugin (PR #23)
 Allow dynamic inline ECU-TEST instances in pipelines (PR #31)
 Allow dynamic inline TEST-GUIDE instances in pipelines (PR #29)

 Use random generated id for linking reports (PR #24)

Release 2.6 (Feb 15, 2019)

Add option to update user libraries (PR #12)
Support generation and re-use of ECU-TEST caches (PR #13)
Allow individual Windows task name (PR #19)

Request API endpoint to check TEST-GUIDE availability (PR #15)
Internal refactoring (PR #7)

Reference filter validation throws ANTLR exception (PR #17)

Updated available ATX settings to TEST-GUIDE 1.64.1
Requires at least TEST-GUIDE 1.55.0

Release 2.5 (Dec 07, 2018)

Allow dynamic TEST-GUIDE configuration in Pipelines
 Add partial JCasC plugin support to manage ECU-TEST installations

Use unique ATX id for linking reports to TEST-GUIDE if available
Improved ECU-TEST version parsing (PR #6)

 Ensured compatibility up to ECU-TEST 7.2.0

Release 2.4 (Jul 30, 2018)

Fixed security issues SECURITY-932, SECURITY-994

Release 2.3 (Jul 20, 2018)

Allow to use TMS credentials from any Jenkins context
Removed existence check of .workspace directory
Improved COM connection stability by updating to JACOB 1.19
 More robust parsing the ATX upload response

Updated available ATX settings to TEST-GUIDE 1.57.1

Release 2.2 (Apr 13, 2018)

Allow additional data types as global constant
Improved COM connection stability with COM timeouts
Evaluate ATX upload info for linking ATX reports

Ensured compatibility up to ECU-TEST 7.1.0
Updated available ATX settings to TEST-GUIDE 1.54.0

Release 2.1 (Feb 16, 2018)

Added publisher for downstream trace analysis

 Disabled COM timeout by default

Fixed ATX report links when using downstream publisher

Bumped Jenkins baseline to LTS 2.60.3

Release 2.0 (Dec 08, 2017)

Added publisher for downstream report generation
Added build variables containing last loaded ECU-TEST configurations
 Added COM timeout to detect non-responding ECU-TEST instance or tool connections

Fixed sporadic errors when setting global constants

Bumped Jenkins baseline to LTS 2.60.1
Requires Java 8 on master and slaves
Ensured compatibility up to ECU-TEST 7.0.0
Updated available ATX settings to TEST-GUIDE 1.49.0