Child pages
  • Job Generator Plugin
Skip to end of metadata
Go to start of metadata

Summary

This plugin adds a new job type "Job Generator" which can generate new projects when executed.

View Job Generator on the plugin site for more information.

Table of content 

Why this plugin ?

This plugin is a good fit for generating pipelines based on parameterized templates. Typical parameter is the branch name or base directory where to execute a job.

It is possible to do it with the built-in build parameters of Jenkins but you have to manage input/output parameters for each job (i.e. duplicate the branch name parameter) if you want to be able to execute a job manually in the pipeline. Moreover some plugins don't expand variables which can be a show-stopper till they get fixed. Last but not least you may want to avoid to pollute the job history with different parameters which make hard to follow trends (i.e. different branch locations) and you need history isolation with different explicit jobs pre-configured with each parameter value.

Of course you don't want to do this by hand as the number of jobs can grow quickly if you have a big pipeline. Many teams work around this by developing external tools to duplicate jobs. There are some plugins available which allow to generate job configurations, you may want to take a look a the Job DSL Plugin.

The Job Generator plugin proposes a different approach by allowing to define job templates in a natural way, there is no need to learn an additional script language.

Features

  • Global and Local Generator Parameters.
  • Powered by the plugins Parameterized Trigger PluginConditional BuildStep Plugin and Flexible Publish Plugin
  • Pipeline generation and deletion in a few clicks.
  • Conditional steps are resolved at generation time.
  • Requires explicit action from the user for critical operations.
  • Should support all free-style job plugins out of the box.
  • Hot creation/deletion of jobs. No need to reload from disk and restart jenkins.

How it works ?

A job generator copies itself when it is executed replacing all generator parameters by the given values. The conditional entries (Conditional BuildStep Plugin and Flexible Publish Plugin) are resolved during the generation so generated job configurations are clean.

By using the Parameterized Trigger Plugin (see example below) you can generate multiple jobs in one pass by calling a downstream job generator multiple times with different parameter values; this way you can fork jobs easily.

Configuration

After installing the plugin you'll get a new entry in the Job creation view.


A new type of job

The job configuration view is the same as a free-style software build job with a few additions:

  • New choices of build parameters:  Generator Parameter, and Generator Choice Parameter.
  • New choices of parameters for the Parameterized Trigger Plugin: Generator Parameter and Current Generator Parameters.
  • Text box entries to provide the name of the generated job as well as its display name (section is at the bottom of the configuration page).
  • Check box to automatically run the generated job once the generation is complete.


Build parameters


Trigger build parameters


Provide a name for the generated job

When a Job Generator is executed an action view is displayed allowing you to define the declared generator parameters and to set several options:

  • Process this job only: Downstream jobs are skipped.
  • Delete jobs: If checked, the jobs are deleted instead of being generated.


Generation Page

Usage

In a nutshell

Define your pipeline once with job generators => Make instances of your pipeline by running the top most job generator (trigger of your pipeline). There is no need to restart the server to use the generated jobs.

Details

Parameters

Parameters are defined using the regular build parameters of Jenkins. New types of build parameters and factories allow to define generation parameters:

  • Predefined Generator Parameter: a key-value parameter.
  • Generator Choice Parameter: a set of predefined key-value parameters.
  • Counter Generator Parameter Factory: it creates generator parameters based on a counter.
  • File Generator Parameter Factory: it creates generator parameters based on a list of parameters stored in a file.

As you can see on the above screenshot, Generator Parameters are splitted into two categories depending on their declaration location:

  • Global: These parameters are declared in the top most jobs of the pipeline (the trigger jobs). They are globally available in the whole pipeline.
  • Local: These parameters are declared in any other jobs which are not top most. They are only visible for the job which declares them. This is useful to fork a generation at a certain point in the pipeline, this is possible by using the Parameterized Trigger Plugin which can call the same Job Generator several times with different values for the local Generator Parameters.

Job Names

The name of a generated job must be provided and it cannot be the same as the Job Generator name or cannot produce a name of a Job Generator.
To let the display name parameter available to the Job Generator, a special configuration item is available in the Additional Configuration section to define the display name for the generated jobs.

Recommended additional plugins

This plugin works best coupled to the following plugins:

Good practices

  • You can prefix the Job Generator names (i.e. name starting with TL_) and put them in a special view with a matching regular expression. Use of the View Job Filters to hide this view to non-admin users.

Advanced Usage

On the fly replacement

It is possible to perform string replacement operations at generation time. There are 2 special forms for string replacement:

  • Asterisk
  • Regular expression (something like sed)

Asterisk form
If you put an asterisk at the end of a variable name then all special characters will be replaced by underscores.
Exhaustive special characters list:  \ / : * ? " < > | 

Example:

FOO = "<Hello::World>"
${FOO*}= "_Hello__World_"

Note: Generated job names are always generated with the asterisk form.

Regular expression form
Regular expression can be specified with an a la sed syntax. Some examples are worth a thousand word:

FOO = "One more hello world example"
${FOO/.*(hello).*/$1} = "hello"
${FOO/\s/_} = "One_more_hello_world_example"

Example/Tutorial

Given a pipeline

A > B > C

where A is the trigger job which defines the following global generator parameter:

  • BRANCH

And C is the last job which defines the local generator parameter:

  • LANGUAGE

Configuration of job A

  • A build parameter with type Generator Parameter and name BRANCH
  • An Execute Shell build step: echo A
  • Generated Job Name
    ${BRANCH}_A
    
  • Generated Job Display Name
    (${BRANCH}) A Trigger
    
  • A publisher Trigger a parameterized build for job B with:
    • a parameter type Current Generator Parameters

Configuration of job B

  • Generated Job Name:
    ${BRANCH}_B
    
  • Generated Job Display Name:
    (${BRANCH}) B
    
  • An Execute Shell build step: echo B
  • A publisher Trigger a parameterized build for job C with:
    • a parameter type Current Generator Parameters
    • a parameter type Generator Parameter: LANGUAGE=Cpp
  • Another_ publisher _Trigger a parameterized build for job C with:
    • a parameter type Current Generator Parameters
    • a parameter type Generator Parameter: LANGUAGE=Java

Configuration of job C

  • Generated Job Name:
     ${BRANCH}_${LANGUAGE}_C
    
  • Generated Job Display Name:
    (${BRANCH}) C (${LANGUAGE})
    
  • An Execute Shell build step:
    echo ${LANGUAGE}
    

Launch a build on job A with BRANCH=prod, the generated jobs are:

prod_A > prod_B > prod_Cpp_C
                > prod_Java_C
(prod) A > (prod) B > (prod) C (Cpp)
                    > (prod) C (Java)

When this pipeline is executed the following merged output is:

~$ A
~$ B
~$ Cpp
~$ Java

Limitations

  • Support for free-style build software jobs only.
  • Job generators cannot use Poll SCM and Time triggers (but it is possible to define a free-style top level job to trigger the generation).

Todo list

  • Check for the possibility to support all kind of jobs.
  • Preview of impacted jobs in the generation view.
  • Batch generation by providing multiple values separated by commas.

Need help ?

The comments on this page are not well suited to ask for help. Please join the  jenkins-user mailing list and ask your questions there. For bug report or new feature request, please open a ticket on JIRA.

Change log

1.23 (will be released soon)

New Feature

Dependencies

1.22 (released April 13, 2014)

Major bug fix

  • [JENKINS-22455] FATAL: Null value not allowed as an environment variable: VARIABLE_NAME
1.21 (released February 15, 2014)

New feature

  • [JENKINS-21238] Ability to run the generated job immediately after generation.
1.20 (released December 29, 2013)

Major bug fix

  • [JENKINS-21072] White screen after submitting job generation (thanks to Alex Artyukhovskii).
1.19 (released December 11, 2013)

Major bug fix

1.18 (released July 16, 2013)

New feature

1.17 is a non-release.

1.16 (released July 14, 2013)

Major bug fix

  •  [JENKINS-18663] Generator triggers the build step using "conditional build step"

Note: Freestyle jobs which are part of a Job Generators hierarchy are not executed anymore at generation time as in 1.15. This should not bother anyone since it should be the expected behavior. If this behavior causes some troubles please post an issue on the bug tracker.

1.15 (released July 4, 2013)

Major bug fix

  •  [JENKINS-17814] hudson.model.FreeStyleProject cannot be cast to org.jenkinsci.plugins.jobgenerator.JobGenerator
1.14 (released May 30, 2013)

Major bug fix

  •  [JENKINS-18146] Support for comma separated fields for downstream jobs

New feature

  • Add a new generation option: Disable jobs. If true, all generated jobs will be disabled.
1.13 (released May 29, 2013)

(warning)   WARNING !! 

This version has a breaking change: GeneratorKeyValueBuildParameters class has been removed. It is replaced by PredefinedGeneratorParameters.

Please DO NOT remove the data in Manage Old Data section (as seen in the thumbnail below), instead you can make a global search and replace in all your config.xml files to replace all GeneratorKeyValueBuildParameters occurrences to PredefinedGeneratorParameters and then restart your Jenkins server.

New features

  • New Counter Generator Parameters for 'call build' build step of Parameterized Trigger Plugin.
  • New File Generator Parameters for 'call build' build step of Parameterized Trigger Plugin.
  • New Predefined Generator Parameters (it replaces the old GeneratorKeyValueBuildParameters).

Major Change

  • GeneratorKeyValueBuildParameters class has been removed.
1.12 (released May 27, 2013)

Major change

Bug Fixes

  • [JENKINS-17862] multi step job generation doesn't work with multiple calls.
  • [JENKINS-18100] Nested call builds in conditional steps are not triggered and their names are not expanded.
1.11 (released May 26, 2013)

New feature

1.10 (released March 16, 2013)

Bug Fixes

  • Fixed special characters in generated job names, the characters  \ / : * ? " < > |  are now replaced by _
  • Fixed support for Parameterized Trigger Plugin build steps (which are no longer listed as downstream projects since version 2.17).

Dependencies

1.09 (released January 17, 2013)

New feature

  • Added support for http://<servername>/job/<jobname>/buildWithParameters end point.
1.08 (released January 8, 2013)

New feature

  • It is now possible to insert a regular job type (i.e. Free-Style Build Software job) in the hierarchy of job generators.

Enhancements

  • Evaluation of conditional blocks from Conditional BuildStep Plugin and Flexible Publish Plugin are performed for the following expressions only:
    • Boolean condition
    • Numerical comparison
    • Regular expression match
    • String match
  • Display a summary of deleted jobs when deleting a hierarchy of generated jobs.

Major change

  • Simplified the options on the generation page to better fit the most common use case:
    • removed "overwrite" option, a generation now always overwrites an existing job.
    • Inverted option to generate downstream jobs. Now the default behavior is to generate downstream jobs and there is an option to only generate the launched job generator.

Bug fix

  • Fixed a race condition when deleting a hierarchy of generated jobs.
1.07 (released 12-12-12)

Enhancement

  • Generator parameters are recursively evaluated.
  • i.e. X="${VAR1}" with VAR1="${VAR2}", VAR2="Hello" and VAR1, VAR2 are generator parameters, X is evaluated to "Hello"
    
1.06 (released December 10, 2012)

Major bug fix

1.05 (released December 10, 2012)

Major changes

Major enhancement

1.04 (released December 5, 2012)

New feature

  • Added a summary in build page showing which job has been created or updated.

Minor change

  • Removed quiet period when a Job Generator is triggered.
1.03 (released December 3, 2012)

New features

  • New generator build parameter entry available: Generator Choice Parameter.
  • New parameterized build trigger entry available: Current Generator Parameters.

Minor change

  • Added 2 specific sections for global and local generator parameters in build action view.

Bug Fix

  • Trigger job names in parameterized trigger section are now correctly resolved when there is more than 1 parameter specified in the Generator Parameter field.
1.02 (released December 2, 2012)

Major changes

  • changed base package and renamed classes which break the backward compatibility with version 1.01.
  • mandatory dependency on Parameterized Trigger Plugin

New feature

Minor changes

  • renamed Template Parameter for Generator Parameter.
  • Revamped UI of generation view in order to be simpler (removed unnecessary Additional parameter block).

Bug fix

  • Fixed generation of fields containing a coma separated list of projects (previous version only generated the first entry in the field).
1.01 (released November 29, 2012)

New features

  • SCM and Time build triggers are now ignored for Job Generator.
  • SCM configuration is now ignored when a Job Generator is executed.
  • Job generators are always executed on the master node.

Minor change

  • Configuration entries added by the plugin are now in their own section at the bottom of the configuration page.

Bug fixes

  • Build is now marked as a failure if the generated job name is empty or equal to a Job Generator name.
  • Generated Job Display Name is now correctly saved.
  • Fixed a side effect when using the Parameterized Trigger Plugin which prevented the predefined parameters from being generated.
1.0 (released November 28, 2012)
  • Initial version

About

18 Comments

  1. Screenshots might make it easier to understand exactly what this plugin does and how it does it.

    1. Added screenshots with some more explanations about how it works.

  2. This plugins seems just what I need, but I really need support for "custom workspace".

    It seems to work just fine if I add it manually to config files, but the parameter is lost after editing the template config.

    Can this be added?

    Also, minor bug: the "Run buildstep before SCM runs" is executed when generating jobs.

    1. Hi Marko,

      I created associated tickets in JIRA, feel free to comment on it to provide more information.

      Thank you.

  3. Hi,

    I try to utilize this plugin but downstream project never gets the predefined generator parameter (in the example: LANGUAGE). I created the sample scenario above but also fail to create the proper jobs.

    The following jobs generated:
    
    (prod) A Trigger
    (prod) B Trigger
    (prod) C (${LANGUAGE})
    

    Jenkins Version: 1.532.1
    Job Generator plugin: 1.20
    Parameterized Trigger Plugin: 2.21

    Can you give me a hint what is wrong?

    Thank you

    1. Hi,

      Are you passing the generator parameters explicitly with either "Current generator parameters" or "Predefined generator parameters" ?

      1. Hi,

        In Job "B" I use both, like the example says:

        Configuration of job B

        A publisher Trigger a parameterized build for job C with:

        • a parameter type Current Generator Parameters
        • a parameter type Generator Parameter: LANGUAGE=Cpp

        Another_ publisher _Trigger a parameterized build for job C with:

        • a parameter type Current Generator Parameters
        • a parameter type Generator Parameter: LANGUAGE=Java
        1. Hi,

          Is the "Generated Job Name" correctly set ? If all settings are correctly set, can you open an issue on JIRA and attach your .xml configurations ?

          Thank you.

  4. Hi,

         I got a potential bug with this plugin(may be).

    When we try to delete the jobs using this plugin, It eventually ended up deleting some other jobs (usually last job which it generated) instead of the one which we want.

    Has anybody else encountered this problem.

    Thanks,

    Vishal

  5. Does anyone know if the Job Generator plugin is compatible with the Join Plugin? I've been experimenting with it and it seems only partially work when dealing with downstream jobs. 

    In my experiment I had one upstream generator job that had two downstream generator jobs. The upstream generator job had a join trigger to start a third generator job when the two downstream jobs finished. The one upstream and two downstream jobs were created successfully but the Join trigger job was not generated. It still referenced in the generated upstream job as it's original job name.

    I'm pretty sure the two are not compatible but I figured I should ask.

    Thanks

  6. I wanted to rerun the same job every 15 minutes only if the windows batch command fails and the script has to rerun for 4 times  after the first failure.

    I am new to jenkins i have installed this plugin it will be very helpful if i can get some help on creating the job in jenkins.

    Regards,

    Shashank

  7. I want to use this plugin, but I'm concerned about whether it is still alive.  What if I become dependent on it and when I upgrade to the next version of Jenkins, this plugin is broken?  Can I get a sign that someone is still supporting this?

    1. @Shannon, I think it might be abandoned. Clicking on the above Jira link gives; The value 'jobgenerator' does not exist for the field 'component'.

      1. Thanks, Anne.  We went ahead and decided to use it.  I've never supported a plugin before, but there are a couple issues with it that should definitely be fixed.  You don't happen to know the process for taking ownership of an abandoned plugin do you? :)

  8. Pretty sad to see this plugin is pretty much abandoned, as it has had no activity for two years, since it's a really useful plugin. Any chance we can get a new build of this plugin from the Git repository? The currently released version has problems with the Flexible Publish plugin, which are solved in the last version from the repository.

  9. Is there a way we can use this plugin to generate a job in a specific folder?

    Also, I noticed that it throws an error when we try to move a generated job to a specific folder?

    Does anyone else face this issue?

  10. Reagan Abraham, you are absolutely right. Very disappointing. You can move the generated job to a specific location but first you must to delete an empty folder that was created during the job generating. This empty folder is created at the same location where the Generator job is placed.

  11. Unfortunately there is no way to somehow configure the Generator in order to tell him which configuration to clone and which not.

    The background:
    I as Admin want to restrict normal users from beeing able to edit/delete the generator itself but allow them to configure a generated Job afterwards.
    I tried to use the Project-based authorization but ofcourse the authorization matrix is cloned, too, during the generation => normal users cannot change/delete the generated Job.

     

Write a comment…