Goals
- To run "like-real" user transactions in Jenkins in order to validate that it will perform the tasks that a normal user would expect to function with each change submitted by a Jenkins developer. This would effectively be the first "real" traffic we get in the essentials instance and it will allow us to be the first ones to know if something went wrong and react fast to impact the least users possible.
Background and strategic fit
Similar to the Acceptance Tests, the goal for Synthetic Transaction tests are to validate key "user scenarios" within Jenkins, which span multiple plugins that provide functionality to users. Such as "Creating a new Pipeline in Blue Ocean" which may exercise aspects of multiple plugins which work together. Unlike Acceptance Tests, these tests are not intended to cover everything but rather a smaller subset of "smoke tests" to rapidly validate whether Jenkins is behaving how an end-user would expect use Jenkins "for real."
Additionally, the long-term goal for Synthetic Transaction tests are that it should be possible to run post-release in a live Jenkins environment, similar to "production synthetic checks" and monitoring for web services.
Assumptions
- Jenkins can drive the testing of itself.
Requirements
| # | Title | User Story | Notes |
|---|---|---|---|
| 1 | Execute a synthetic test in a pre-existing Jenkins | An existing Jenkins instance must be capable of executing synthetic tests. That is to say, the tests must not require a "full setup of a Jenkins Under Test." | |
| 2 | No side-effects | Execution of a synthetic check must have no side-effects on the Jenkins instance, i.e. idempotent. | |
| 3 | A synthetic test must be fast | Each synthetic test must be able to complete in a matter of seconds or under a couple minutes, such as not to adversely tax the system. | |
| 4 | A synthetic test must provide consistent results | Each test should not have any flakiness, the results of a test should be very trustworthy to Jenkins developers. | |
| 5 | Each post-release synthetic test must be useful to a user | Post-release synthetic tests should appear, or be understood, by end-users as tutorials, or examples of how to use Jenkins Pipeline (for example). | |
| 6 | Each synthetic test should be runnable pre- and post-release | The Pipeline for Jenkins should execute synthetic tests at the end of the Pipeline, but also the tests should be able to be executed post-release/post-upgrade. |
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|
Do we want to have one test or several? Do we want to do it via UI or via CLI? | We will need several test cases, one for each key user transaction |
| Cleanup of the test environment? | ??? |
| Will the listed User Transactions catch "known" past-issues released to the Jenkins user base. | ??? |
| For Jenkins Developers, are many of their initiatives able to be validated with immutable (no side-effect) synthetic tests. | |
| How are synthetic tests updated? |
User Transactions
- Execute a Pipeline with JUnit test results and an Artifact
- Execute a (Declarative) Pipeline with an email notification in the post directive
- Execute a Multibranch Pipeline which must load a
Jenkinsfilefrom GitHub in order to execute.