Jenkins : 2.0 Pipeline as Code

Please use epic ticket for comments, feedbacks, etc.

Introduce a new subsystem in Jenkins that:

  • lets you design a whole pipeline, not just a single linear set of tasks
  • stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
  • makes it automatic to set up new pipelines when Jenkinsfile is added
  • differentiates multiple branches in the same repository

This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

Overview

(This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

  1. In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
  2. In one of the repository in the chosen org, create a Jenkinsfile that defines your pipeline (or just a build process to start with.) This file is actually a Pipeline script. See Pipeline tutorial for details.
  3. Jenkins will automatically recognize this branch and create appropriate jobs by itself.
  4. Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
  5. When the branch is destroyed in the repository or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins as well automatically.

In this use, there'll be a lot of Jenkinsfile in various branches & repositories. To keep them DRY, various mechanisms will be provided to promote reuse of Pipeline scripts, such as this.

For more details, see Unknown User (jglick)'s slides and video recording that talks about this feature (in particular, where the demo starts.) There's also docker-based demo that you can play with on your own.

Contents

  • Thus, this feature should be made available out of the box by default, unless users opt out (2.0 Out-of-the-box experience)
  • Provide better Pipeline visualization out of the box (JENKINS-31154)
  • Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
  • Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

Internals

  • The execution engine of this is Pipeline Plugin (see JENKINS-26129)
  • Pipeline Multi-branch project type defines a new kind of folder that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroied in the repository.
  • Organization folder that defines a new kind of folder that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "Pipeline Multi-branch project" as repositories are created/destroyed in the group.
  • Branch API plugin and SCM API plugin defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
  • GitHub Branch Source Plugin implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.