Skip to end of metadata
Go to start of metadata
Target release
Epic
Document status
DRAFT
Document owner
Designer
Developers
QE

Goals

  • To run "like-real" user transactions in Jenkins in order to validate that it will perform the tasks that a normal user would expect to function with each change submitted by a Jenkins developer. This would effectively be the first "real" traffic we get in the essentials instance and it will allow us to be the first ones to know if something went wrong and react fast to impact the least users possible.

Background and strategic fit

Similar to the Acceptance Tests, the goal for Synthetic Transaction tests are to validate key  "user scenarios" within Jenkins, which span multiple plugins that provide functionality to users. Such as "Creating a new Pipeline in Blue Ocean" which may exercise aspects of multiple plugins which work together. Unlike Acceptance Tests, these tests are not intended to cover everything but rather a smaller subset of "smoke tests" to rapidly validate whether Jenkins is behaving how an end-user would expect use Jenkins "for real."

Additionally, the long-term goal for Synthetic Transaction tests are that it should be possible to run post-release in a live Jenkins environment, similar to "production synthetic checks" and monitoring for web services.

Assumptions

  • Jenkins can drive the testing of itself.

Requirements

#TitleUser StoryNotes
1Execute a synthetic test in a pre-existing Jenkins

An existing Jenkins instance must be capable of executing synthetic tests. That is to say, the tests must not require a "full setup of a Jenkins Under Test."

 
2No side-effectsExecution of a synthetic check must have no side-effects on the Jenkins instance, i.e. idempotent. 
3A synthetic test must be fastEach synthetic test must be able to complete in a matter of seconds or under a couple minutes, such as not to adversely tax the system. 
4A synthetic test must provide  consistent resultsEach test should not have any flakiness, the results of a test should be very trustworthy to Jenkins developers. 
5Each post-release synthetic test must be useful to a userPost-release synthetic tests should appear, or be understood, by end-users as tutorials, or examples of how to use Jenkins Pipeline (for example). 
6Each synthetic test should be runnable pre- and post-releaseThe Pipeline for Jenkins should execute synthetic tests at the end of the Pipeline, but also the tests should be able to be executed post-release/post-upgrade. 

 

Questions

Below is a list of questions to be addressed as a result of this requirements document:

QuestionOutcome

Do we want to have one test or several? Do we want to do it via UI or via CLI?

We will need several test cases, one for each key user transaction
Cleanup of the test environment????
Will the listed User Transactions catch "known" past-issues released to the Jenkins user base.???
For Jenkins Developers, are many of their initiatives able to be validated with immutable (no side-effect) synthetic tests. 
How are synthetic tests updated? 

User Transactions

  • Execute a Pipeline with JUnit test results and an Artifact
  • Execute a (Declarative) Pipeline with an email notification in the post directive
  • Execute a Multibranch Pipeline which must load a Jenkinsfile from GitHub in order to execute.