Child pages
  • Priority Sorter Plugin
Skip to end of metadata
Go to start of metadata

This plugin allows Jobs to be prioritised based on Queue Strategies and Priority Strategies.

View Jenkins Priority Sorter Plugin on the plugin site for more information.

Description

This plugin adds the ability to assign different priorities to jobs: the lower the priority a job has, the sooner the job will run.

This can be very helpful, when one wants to add low priority jobs, but wants to have higher-priority jobs run first. This is especially true when hardware is limited, or when there are different groups of jobs that should share resources (equally).

Fundamentals

The plugin is build around some basic concepts:

  • Queue Strategy
  • Job Group
  • Job Inclusion Strategy, and
  • Priority Strategy.

First, the Queue Strategy describes how the priority of a job is interpreted. It will translate the job's assigned priority to a corresponding weight in the queue. The queue is, then, sorted based on the assigned weight.

With the Priority Sorter, you will need to group your jobs into Job Groups. A Job Group's Job Inclusion Strategy devides whether a specific job should be included in the group.

The priority of the job is then set by the Priority Strategy, and each Job Group can have any number priority strategies.

When a new job is queued, the following will happen:

  • The Priority Sorter will go though the configured Job Groups from top to bottom.
    • When it finds a Job Group where the job is to be included, it will look though the Priority Strategies from top to bottom
      • When it finds a Priority Strategy that matches its criteria, it will use this strategy to assign a priority the the Job.
  • The Priority Sorter will then consult the Queue Strategy to translate the priority to a weight, i.e. a position in the queue.

Queue Strategies

There are three included Queue Strategies, each of which will sort the Queue differently.

Absolute

Each job will be assigned a priority, and, the lower the priority, the sooner the job will be run.

Fair Queuing

Each Job will be assigned a priority, but the queue will try to share the resources equally over different priorities in a round-robin fashion.

Weighted Fair Queuing

The same as Fair Queuing, but jobs with a lower priority will be run more frequently those with higher a priority. Remember, that the lower the priority assignment, the higher the importance.

Job Inclusion Strategies

This setting describes which jobs are included in a Job Group, and is relatively self explanatory. It is worth mentioning the "Jobs Marked for Inclusion" setting. When this is selected, you are requested to enter a name of the Job Group. This name will then be an available option on each job, providing an alternative "bottom-up" way to group jobs, rather than the default "top-down" approach of the other strategies.

Priority Strategies

It is possible to assign the priority, based on different Priority Strategies.

This setting describes what priority a job should have, and is self explanatory. It is worth mentioning the the "Take the Priority from Property on the Job" setting. When selected, a drop-down - where you can select a priority - appears on all projects that would generate jobs to be included in this group.

Main Configuration

The plugin's main configuration options are available on the Jenkins configuration page. Configuration options for Priority Strategy are available on its own page, accessible from the root actions menu.

On the main configuration page, you can select the number of priorities you would like to use, as well as a default priority to use where no other priority could be assigned.

(warning)  Please note that lower number means higher priority! See details on the screen, for the specifics on each strategy. 

Assigning Priorities

On the main menu, you will find the link to a page where you can assign priorities to each Job

On this page you will be able to assign jobs to groups, and priorities to jobs.

(warning)  Please note that all matching is done top to bottom by first match

You can limit access to this functionality to administrators, by checking the appropriate check-box on the main configuration page.

Run Exclusive

If a job from a Run Exclusive job group gets started, jobs from other groups will remain blocked, and will not get executed until all jobs from the Run Exclusive job group are complete, regardless of priority.

Using the View Column

Since a job can get a different priority each time it is started, the view column cannot show the "correct" priority for the Job.

The column will show the priority used the last time the job was launched, and, if the job has not been started yet, the column will show Pending.

Notable changes and upgrading

Upgrading from 1.x

Version 2.x is a complete rewrite of the plugin, but still supports running in version 1.x compatibility mode. However if you are satisfied with the functionality of 1.x, there is little point in upgrading.
Upgrading from version 1.x to version 3.x will remove all 1.x configurations. Therefore, if you need to keep configured values, upgrade to 2.x, first, switch to Advanced Mode, and then upgrade from 2.x to 3.x.

Upgrading from 2.x

In version 3.x, the option "Allow priorities directly on Jobs" has been removed in favor of the Priority Strategy "Take the priority from Property on the Job" (see above). Legacy Mode is removed (see above regarding upgrading from version 1.x).

Troubleshooting

To get some inside information on how, and why, a certain job gets a certain priority, you can turn on some extra logging.

  • Logger: PrioritySorter.Queue.Items
    • To get more info on the assigned priorities, and state transition of the items in the queue, set the log level to FINE.
    • To get more info on how the jobs are matched to job groups, and rules to get the priority, set the log level to FINER.

To get logging on when the Queue Sorter is active log, use:

  • Logger: PrioritySorter.Queue.Sorter, with the level set to FINE.
    • To see all items sorted (the queue) by the Queue Sorter, set the log level to FINER.

Changelog

Version 3.6.0 (2018-01-12)

  • PR #42 - Make plugin compatible with class serialization whitelists in Jenkins Core 2.102 and above (JEP-200)
    • Warning: the change fixes the base class of the SorterStrategy extension point. If you implement it in plugins, it is recommended to update the dependency
  • PR #40 - Null safety in some plugin methods (e.g. Strategy#getPriority()

Version 3.5.1 (2017-06-18)

  • JENKINS-41419 - Fix sorting of queue items when `sortAsInQueueSince` is set
  • JENKINS-42584 - Take priorities of upstream jobs into account when scheduling downstream jobs
  • PR #38 - Prevent NullPointerException in PriorityConfigurationPlaceholderTaskHelper when restarting Jenkins with pending jobs
  • JENKINS-44014 - Prevent NullPointerException in AdvancedQueueSorter#onLeft() when item is missing in the queue
    • Instead of exceptions, there will be warning messages in Jenkins logs with diagnostics info
  • JENKINS-44014 - Prevent ClassCastException if owner of the Pipeline placeholder task is not a Job
    • In such case a default priority will be set. The issue will be logged on the FINE level

Version 3.5.0 (2017-01-18)

Fixes : JENKINS-40284 Fix blocked weights with Fair Queuing strategies
Fixes : JENKINS-36570 Add Pipeline support

Version 3.4.1 (2015-09-29)

Fixes : JENKINS-37644 Update Folders Plugin dependency to fix the compatibility issue

Version 3.4 (2015-06-06)

Fixes : JENKINS-28621 Jenkins jobs get held up in queue waiting for available executors, even though all are available

Version 3.3 (2015-05-26)

Fixes : JENKINS-28462 Priority selection on Job is "always" shown
Fixes : JENKINS-28461 Control Node usage based on Priority
Relates : JENKINS-23640 Validate regular expression and report matching jobs on Job Priorities page

Version 3.2 (2015-05-18)

Fixes : JENKINS-24962 Cannot assign a JobGroup to a Nested (sub) View
Fixes : JENKINS-28280 Wrong Job Group is shown in the selector (one more place) (Thanks to Kyrremann)

Version 3.1 (2015-05-12)

Fixes : JENKINS-28195 Jenkins not scheduling any jobs after upgrade to Priority Sorter 3.0 but giving exception
Fixes : JENKINS-28280 Wrong Job Group is shown in the selector (Thanks to Kyrremann)
Fixes : JENKINS-28359 Remove support and conversion from Legacy Mode

Version 3.0 (2015-05-02)

Fixes : JENKINS-21337 Add support for cloudbees-folders
Fixes : JENKINS-21356 Add support for using JobProperty to join a Job to a JobGroup
Fixes : JENKINS-23538 Remove support for Legacy Sorter
Fixes : JENKINS-23552 Replace "Allow priorities directly on Jobs"
Fixes : JENKINS-23557 Add matrix child jobs to the front of the queue
Fixes : JENKINS-27966 Priority Sorter must not require matrix-project

Version 2.12 (2015-04-15)

Fixes : JENKINS-27957 NullPointerException in AdvancedQueueSorter

Version 2.11 (2015-04-14)

Fixes : JENKINS-27770 AdvancedQueueSorter call to sort violates the comparison contract

Version 2.10 (2015-04-14)

Never released - maven hickup

Version 2.9 (2014-10-08)

  • Fixed NPE happening at startup [X] (Thanks to christ66)
  • Added description field to JobGroups [X] (Thanks to olivergondza)

Version 2.8 (2014-06-17)

Fixes : JENKINS-23462 Sectioned views are not evaluated for job priority

Version 2.7 (2014-06-16)

  • Adds some more logging about queue contents, see Troubleshooting above
  • Performance enhancement when updating the configuration [X] (Thanks to ndeloof)

Fixes : JENKINS-23428 Jobs in NestedView (ViewGroup) are not correctly found

Version 2.6 (2014-01-11)

Fixes : JENKINS-21310 CCE when a Queue.Task was not a Job
Fixes : JENKINS-21316 PrioritySorter wrongly assumes Queue.Task is Job
Relates: JENKINS-21314 Stack trace displayed on web page when attempting to configure PrioritySorter plugin

Version 2.5 (2014-01-08)

Fixes: JENKINS-21289 Item Logging causing NPE
Fixes: JENKINS-21284 Add some logging to show what the sorter is doing
Fixes: JENKINS-21204 Add ability to boost recently failed Jobs

Version 2.4 (2014-01-02)

Fixes: JENKINS-21173 Anonymous Users Can Configure Priorities
Fixes: JENKINS-21119 Extend loggning to show assigned priority
Fixes: JENKINS-21103 "Priorities are assigned top down by first match" does not work correctly (Thanks to Adam Gabryś for debugging help)

Version 2.3 (2013-12-20)

  • introduces the Run Exclusive Mode to deal with JENKINS-11997
  • adds a new Priority Strategy to give Jobs the same priority as a UpstreamJob
  • some performance enhancements
  • hopefully fixes the deadlock issue in JENKINS-21034
  • fixes the priority assignment issue on main configuration page

Fixes: JENKINS-11997 Consider jobs in the Executors for priority sorter
Fixes: JENKINS-21034 Jenkins Startup Deadlock - QueueSorter.installDefaultQueueSorter and Queue.init
Fixes: JENKINS-20995 Default Priority always shows 1-5

Version 2.2 (2013-12-05)

Fixing bug that made the plugin switch to Advanced mode even though Legacy (1.3) data was present in the system. (Thanks to help from Matthew Webber)

Fixes: [JENKINS-8597] Deal with matrix builds better

Version 2.1 (2013-12-04)

Fixes bug that mapped all Jobs to all Views/JobGroups.

Version 2.0 (2013-12-02)

Introducing advanced queueing features with possibility to selected different strategies for how priorities are assigned and how the queue is sorted.

Thanks Oleg for testing, reviewing and helping out.

Version 1.3

Removed view column from default view (Thanks to work from larrys)

Version 1.2

Added View column to easily compare priorities between jobs (Thanks to work from cjo9900)

Version 1.1

Fixed a potential NPE when using the plugin on existing jobs without setting a default priority.

Version 1.0

Initial Release

28 Comments

  1. It's not clear what represents 'higher priority'.  Is it higher numbers or lower numbers?

    Also what is the priority of builds which have no priority set?  Will they have higher or lower priority that a build with a priority value set?

    1. Hello,

      a try at an answer:

      It's not clear what represents 'higher priority'. Is it higher numbers or lower numbers?

      Ok, the readme seems to imply that higher priority (meaning higher numbers) are higher priority (wink)
      And RTFS shows this:

      @Extension
      public class PrioritySorterQueueSorter extends QueueSorter {
      private static final class BuildableComparitor implements
      Comparator<BuildableItem> {
      public int compare(BuildableItem arg0, BuildableItem arg1) {
      // Note that we sort these backwards because we want to return
      // higher-numbered items first.
      Integer priority1 = getPriority(arg1);
      return priority1.compareTo(getPriority(arg0));
      }
      

      Also what is the priority of builds which have no priority set?

      Also RTFS (wink)

      public class PrioritySorterDefaults {
      final static int defaultPriority = 100;
      static int getDefault() {
      return defaultPriority;
      }
      }
      So the default is 100. (At least on 8. August 2012)
      

      Regards

      Martin

      1. Since when was RTFS an acceptable response to this sort of enquiry???  Not everyone who wants or needs to use a tool has an indepth knowledge of the coding for the tools, and even if I do happen to, why would I want to waste my time checking out and reading through code to find the answer to basic usage information?

  2. Would be great to prioritize based on last build duration in order to (mainly) execute non-test (build-only) jobs first.

  3. Wish List: It would be nice to be able to pick a job out of the Queue and boost its priority for just this run.

    Once a job is setup, the priorities generally don't change.

    However, the exception to this is somebody watching the Queue and debugging TODAY.  They will boost the priority for their debug session, but then almost never set it back to where it was.  It would be nice if the priority could be adjusted for one time use only and would then revert to its original value.

    Frank

    1. GREAT plugin BTW: Thank you

  4. Hi, the priority does not seem to work for multi-configuration job (jenkins version 1.523 / priority sorter plugin version 1.3). Whatever priority is chosen the order of jobs remains the same.

    Is multi-configuration job supported? Is it a bug?

    Robert  

    1. Hi Robert,

      Multi Configuration (MatrixBuild) is supported from version 2.2 - just recently released.

      /Magnus

  5. Is there support for the folder plugin? Ie setting priorities on a folder basis?

    1. Please have a look at the 3.0 PR (https://github.com/jenkinsci/priority-sorter-plugin/pull/16).

      I unfortunately have very limited time atm so it will take a while before this will get released.

      If you can beta-test it would be great!

      1. Thank you for your reply.

        I'll see if I can find some time myself! I'm not sure if the current functionality meets our needs, but I'll keep you updated

  6. What is the "default" priority notionally applied to jobs for which this plugin is not active (not ticked)?

    Does a priority of 5 beat a non-prioritised job?

    1. The value of default is set on the main configuration page.

  7. I have a setup where I am using the multijobs. So from a couple of different starting points, I run the same subset of jobs.So in the example below, my tests are run from a multijob A1, that starts all of the B jobs.

    A1 -> B1,B2,B3...BN

    Master1 -> A1

    So if Master1 is started 5 times, I want the test jobs, the B's, to take priority then as my test nodes become free start loading the jobs from the next instance of Master1.

    I want them to have a priority among themselves, but also I want one set of these jobs to have a higher priority then future ones. I tried using the priority build parameter, setting the parameter to be 52+($

    Unknown macro: {BUILD_NUMBER}

    *100). For the instance I was looking at the BUILD_NUMBER is 148.

    That did not seem to work for me, as the priority shows as 52. In the configuration, I have the number of priorities set to 100, does that force a limit on the calculated priority? Do I need to set that to 100,000?

    1. Hi,

      I have not used the Multijob plugin so I'm not sure how the PrioritySorter behaves in combination with it.

      I'm not really sure I understand how you want the priorities to behave - do you want the sub-jobs to "inherit" the position in the queue?

      In any case I suggest that you ask on the jenkins-users mailing list and see if someone else has any experience.and also add an item to Jira if you have any features you like to add to the plugin. 

      Thanks,

      /M

      1. What I am trying to do is make the priority offset by the parent's build ID.

        So the first run of A1, is 1. so if the priority of Bn among the B jobs is 'N', then when Bn gets started, it is given a priority of 1000+N.

        Therefor the 12th run of A1, job Bn gets the priority of 12000+N.

        1. Well, that would be the same as it works with matrix where sub-jobs will have the same queue-place as the master job I would say. Currently this is not the way it will work, you will need a multi-job specific extension to priority-sorter do achieve this.

  8. I was wondering if there are some documentation on how to configure Puppet and the Priority Sorter Plugin.

    On the legacy version we used:

    <hudson.queueSorter.PrioritySorterJobProperty plugin="PrioritySorter@1.3">
          <priority><%= priority %></priority
    </hudson.queueSorter.PrioritySorterJobProperty>

    but my guess is that this is not working any more, as we moved on to advanced and using JobGroups.

    1. To answer my own question.

      When making changes to a job the changes are written to a config.xml which you can find under your jenkins/jobs/<job/> folder. For me it was changed from what I wrote in the previous post to this:

      <jenkins.advancedqueue.priority.strategy.PriorityJobProperty plugin="PrioritySorter@3.0">
            <useJobPriority>false</useJobPriority>
            <priority>-1</priority>
      </jenkins.advancedqueue.priority.strategy.PriorityJobProperty>
      
      <jenkins.advancedqueue.jobinclusion.strategy.JobInclusionJobProperty plugin="PrioritySorter@3.0">
            <useJobGroup>true</useJobGroup>
            <jobGroupName>Top Priority</jobGroupName>
      </jenkins.advancedqueue.jobinclusion.strategy.JobInclusionJobProperty>
  9. Make cloudbees-folder optional. For now we are using 3.14 version because guys from cloudbees removed environment variables from 4.0 version and moved it to folder plus version :(. Can you make 3.14 version as required? to not force upgrade during upgrading plugin?

  10. Hi, just a suggestion: I think it would be helpful if the text under *Fundamentals* above is reviewed. It has a number of typos that obscure the meaning. In particular, this sentence is unclear: "With Priority Sorter you will need to group your Jobs into Job Groups, if a Job should be included in a specific Job Group is decided by the Job Inclusion Strategy assigned to the Job Group."

  11. Installed on Jenkins 1.643 and my priorities are completely ignored.

    edit: sorry, rebooting server fixed issue.

  12. hi,
    need your help to understand something..
    we downloaded this plugin in order to able to work with priority.
    in our case: we have 2 jobs (B & C) which is running right after job A. (B & C are starting by upstream project of A, which means once A is completed B & C are starting auto')
    the issue is that project A will not start 'till A & B are finished(both of them) and we have cases that B is running and C just finished so project A will not start because B is still running. in order to run A we need C & B not to be running, we need it to wait in queue.
    so we wanted to apply this priority plugin in order to give A the priority to run first.. and if its sees B/C in the wueue these jobs will wait.. but currently its not working for us..
    so we think we didn't used this configuration correctly.
    can you please suggest??:)
    hi,

    need your help to understand something..

    we downloaded this plugin in order to able to work with priority.

    in our case: we have 2 jobs (B & C) which is running right after job A. (B & C are starting by upstream project of A, which means once A is completed B & C are starting auto')

    the issue is that project A will not start 'till A & B are finished(both of them) and we have cases that B is running and C just finished so project A will not start because B is still running. in order to run A we need C & B not to be running, we need it to wait in queue.

    so we wanted to apply this priority plugin in order to give A the priority to run first.. and if its sees B/C in the wueue these jobs will wait.. but currently its not working for us..

    so we think we didn't used this configuration correctly.

    can you please suggest??:)

    1. You cannot use the Priority Sorter for job dependency management - I think that the conversation in Jira can help you understand: https://issues.jenkins-ci.org/browse/JENKINS-32266

  13. I have two pipeline jobs. Both simply execute "sleep 30", and exit.

    Job 1 is not in any priority group.

    Job 2 is in an "asap" priority group.

    Priority plugin configuration is all default with a Job Group called "asap" with "Jobs to include" set to "Job and Folders marked for inclusion".

    Priority is set to 1 (most important per documentation) for the group "asap".

    I have a node, let's call it "node1". Both pipelines can only run on "node1".

    I trigger job 1 three times in succession.

    I quickly go and trigger job 2 once.

    Now, job 1 is executing the first build, and there are 2 more builds from job 1 in queue, and 1 build in queue from job 2.

    Expectation: Once first build of job 1 finishes, job 2 skips the line and gets the node1.

    Reality: All job 1 builds are run in the order they were requested, and the job 2 which was supposedly the highest priority, runs the last.

    Switching priority group's priority number to 5 (biggest value by default) does not change this outcome.

    What am I doing wrong?

    Does the plugin not work with pipelines? (Issues closed indicate otherwise?)

     

    Jenkins 2.82

    Plugin 3.5.1

     

  14. How do we use this as part of the pipline as code?

  15. Hello Everyone,

    New Year Wishes (smile)

    I am not able to sort the jobs as per the priority using this plugin.

     

    My Requirement

    Jobs in Build Queue should sort as per the priority

    Any new job comes with priority/higher priority should run the job next on lockable resource where it can run.

    Using lockable resources in the Environment 


    Configuration I DID

     

    Priority Sorter
     Strategy
     
     AbsoluteLower priority is scheduled before higher priority. 
     Number of Priorities
       
     The Update will do a linear mapping from your current priorities to the selected range. 
       
     Default Priority
       
       
     
       
     Only Admins can edit Job Priorities 
       
     Check if only Administrators should be allowed to view and edit the Job Priorities

     

     

     

    Job Priorities

    Assign Priorities by creating JobGroups. Priorities are assigned top down by first match. 

     

      
    JobGroup
     Description 
       
       
     Jobs to include 
     
     
     
       
     Priority 
       
     Run Exclusive (experimental) 
       
     
      
    Priority Strategy
     Use Priority from Build Parameter 
     Build Parameter Name 
       
     
       
      
     
       

     

     
       
      
     
       

     

     

    In Jobs using the parameter to assign the priority

       
     
    String Parameter
       
     Name
       
       
     Default Value
       
       
     Description
    [Plain text] Preview 
       
       

     

     

    Plugin behavior Observed

    Jobs are running FIFO even after above configuration and priorities in jobs

     

    Version 

    Jenkins 2.6

    Plugin 3.5.1

     

    Please share your feedback. Thank you

Write a comment…