Child pages
  • Gearman Plugin
Skip to end of metadata
Go to start of metadata

This plugin uses Gearman to support multiple Jenkins masters. 

Plugin Information

View Gearman on the plugin site for more information.

** This project is hosted on git.openstack.org **

Summary

We on Openstack infrastructure team use Jenkins extensively. Our jenkins servers, at peak load, runs 20,000+ jobs per day.   At that load we require many jenkins slaves (900+) to process all of those build jobs.  We have found that our requirement was pushing Jenkins beyond it's limits therefore we've decided to create the Gearman Plugin to support multiple Jenkins masters.  The gearman plugin was designed to support extra slaves, allow load balancing of build jobs, and provide redundancy.  

Jenkins core does not support multiple masters.  You can setup multiple Jenkins masters but there is no coordination between them.  One problem with scheduling builds on Jenkins master (“MasterA”) server is that MasterA only knows about its connected slaves.  If all slaves on MasterA are busy then MasterA will just put the next scheduled build on the jenkin server queue.  Now MasterA needs to wait for an available slave to run the build.  This will be very in-efficient if your builds take a long time to run.  So.. what if there is another Jenkins master (“MasterB”) that has free slaves to service the next scheduled build on the server's queue?  Your probably saying.. “Then slaves on MasterB should run the build instead of waiting for slaves on MasterA to run them”, then I would say "good thought!".  However MasterB will never service the builds on MasterA's queue.  The client that schedules the builds must know about MasterB and then schedule builds on MasterB. This is what we mean by lack of coordination between masters. This  gearman-plugin attempts to fill the gap.

This plugin integrates Gearman with Jenkins and will make it so that any Jenkins slave on any Jenkins master can service a job in the queue.   This plugin will essentially replace the Jenkins (master) build queue with the Gearman job queue.  The job will stay in the Gearman queue until there is a Jenkins node (master or slave) that can run that job.  The gearman job queue is shared by multiple jenkins masters therefore gearman can hand out jobs to the next available slave on any jenkins master.

Features

  • High availability(ish).  When one master goes down the other master(s) will continue to execute builds however the in flight jobs on the downed master will be lost.
  • Slaves are (by default) always shared between masters.  The only way to un-share is to offline or disconnect a slave.
  • Horizontal scalability.  Just continue to add more jenkins masters to distribute the job load between masters.
  • Gearman jobs can start a jenkins build
  • Gearman jobs can stop or abort a jenkins build
  • Gearman jobs can change a build description
  • Gearman jobs can pass in parameters to jenkins builds
  • Gearman jobs can automatically set a slave to offline after running a build
  • Gearman is aware of Jenkins project status: meaning that the gearman plugin will register/unregister projects when the project is enabled or disabled. 
  • Gearman is aware of slave status: meaning thatthe  gearman will register/unregister slaves when a slave is set online/offline and connected/disconnected.
  • Plugin reloads on jenkins restart: meaning that when jenkins restarts the gearman worker threads are automatically restarted and reconnect to a gearman server. 

Known Issues

  • Adding or removing executors on nodes will require restarting the gearman plugin.  This is because Jenkins does NOT provide provide a way to listen for changes to executors therefore the gearman plugin does not know that it needs to re-register functions due to executor updates.
  • The gearman plugin does NOT support Jenkins Matrix Projects.  The gearman plugin relies on project and node labels to register functions correctly.  Matrix projects use labels much differently than the freestyle and maven projects.
  • Since Jenkins 1.651.2, build parameters must explicitly defined in the Jenkins job or whitelisted using system parameters, else they are strip by Jenkins. See https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2016-05-11 and this plugin issue: https://issues.jenkins-ci.org/browse/JENKINS-34885

Getting Started

This assumes some familiarity with Jenkins and Gearman

Install

  • If you don't already have a Gearman server up and running somewhere you should install one.  Theoretically the plugin should work with any gearman server, but we've only used and tested it with the the python gear package.  Install this gearman implementation and run the server.  NOTE: the python gear implementation is only supported on Linux.  Specifically we run on Ubuntu. 
  • Install the Gearman plugin like any other Jenkins plugin, refer to the Jenkins documentation.  You can also get the plugin directly from the Jenkins CI Repository
  • After installation the Gearman plugin the configuration should appear in the Jenkins global configuration page.  Click on the help bubbles if you need additional help with the configuration.  You should test the connection to your Gearman Server before saving your configuration.  Select the 'Enable Gearman' checkbox and click save button will immediately start the gearman workers on the Jenkins server.  

Configuration

Workflow

Starting the Gearman workers:

  1. When the gearman plugin is enabled a gearman worker threads are spawned for each executor on the master and slave nodes.  We'll call these "executor worker threads". Each executor worker thread is associated 1:1 with an executor on a jenkins node.
  2. We spawn one more  Gearman worker thread to handle job management (i.e. abort job/update description/etc..).  We'll call it the "management worker thread" and it will register a "stop:$hostname" and "set_description:$hostname" function with the gearman server.  We use these functions to manage jenkins builds.  
  3. The gearman plugin will register gearman a function for each Gearman executor depending on the projects, labels and nodes that have been setup on the Jenkins master. You can check the registered gearman functions using the administration protocol.  It should look something like this..

Notes:

  1. Red text denotes gearman admin commands
  2. Blue text denotes gearman workers.  There is a default manager worker for the master and an executor worker for a jenkins executor on master.  There are two gearman executor workers for oneiric-668599 slave (exec-0 & exec-1).  These executor workers map to two jenkins executors on the oneiric-668599 slave.  
  3. Functions like "build:guava:ubuntu" map to build:$projectName:$nodeLabel"

Here's the corresponding Jenkins master UI:

Sample Clients

A gearman client can be written in any language.  Here are a few sample clients that work with this plugin

  • java client is a simple client included with jenkins-plugin.hpi

Running a Jenkins build

To  execute a Jenkins job the gearman client just needs to provide the Gearman hostname, port, function, and UUID to start a jenkins build.   

Example:  

python gear_client.py -s MyGearmanServer --function=build:myProject \
    --params='{"OFFLINE_NODE_WHEN_COMPLETE":"false","param1":"moon","param1":"sun"}'

Stopping/aborting a jenkins build

A Gearman request can stop/abort a jenkins build.  

Example: 

python gear_client.py -s MyGearmanServer --function=stop:MyGearmanSever \
    --params='{"name":"myProject","number":"130"}'

The job is stopped differently depending on the current state of the job.  The table below explains the state, transitions and when cancellations happen.

State

Transitions

Cancellation

Gearman queue

Sending a job request to gearman puts it on the gearman queue

the job is removed from the gearman queue

Jenkins queue

jobs on the gearman queue will transition to the jenkins queue

the job is removed from the Jenkins queue

Jenkins executor

job on the jenkins queue transition to the jenkins executor to run

the build is aborted while on the jenkins executor

Updating a build description

You can send a gearman request to update a build's description.  To do this you pass in the following parameters: name of project, build number, description.  

Example:  

python gear_client.py -s MyGearmanSever --function=set_description:MyGearmanSever \
    --params='{"name":"myProject","number":"105","html_description":"<h1>My New Description</h1>"}'

Set slave to offline after a build completes

Our infrastructure employees many 'single use slaves' so what we like to do is run a job and then immediately set the slave offline.  You can do this by passing in the parameter OFFLINE_NODE_WHEN_COMPLETE.

Example:  

python gear_client.py -s MyGearmanSever --function=build:myProject \
    --params='{"OFFLINE_NODE_WHEN_COMPLETE":"true"}'

Configuring Multiple Jenkins Masters

To configure the gearman plugin to work with multiple Jenkins masters you will need to do following:

  • Install the gearman plugin on each Jenkins Master and configure it to connect to the Gearman server (steps above).
  • Create the exact same jobs on each Jenkins master.  The Jenkins Job Builder application is very handy for this purpose and it's what we use.

Now multiple Jenkins masters will be able to service the same jobs.

The typical workflow for this configuration is something like this:

  1. The Gearman workers, running on the Jenkins Masters, are waiting to service the configured jenkins jobs 
  2. A Gearman client submits a request to the Gearman server to run a job.
  3. The Gearman server tells the Gearman worker(s) (on a Jenkins Master) to execute the job(s).  
  4. The first Gearman worker that can service that request will execute the job.  If all workers are busy then the request is placed on the gearman queue to be processed when a gearman worker is available.
  5. The Gearman worker(s) continuously pull jobs off of the Gearman queue and execute each job.  
  6. The Gearman worker reports the job result to the Gearman server when complete.
  7. The Gearman server reports the job result back to the Gearman client.
  8. Loop back to step 1.

Configure logging

Instructions to make the gearman plugin send log messages to the Jenkins logger:

  1. goto http://host:8080/log/levels
  2. add "org.gearman.session.logger" with level "WARNING"
  3. goto http://host:8080/log/all

Now you should see logs from gearman plugin.

Plugin In Action

Plugin In Production

The above images just show how the plugin might work in a simple case.  To see the plugin used in production check out openstack jenkins servers, yes that's servers with an s:

  • jenkins01 - we use this master to run operational jobs
  • jenkins02 - we use this master to run openstack project builds
  • jenkins03 - this is essentially a mirror of jenkins01.

All of the above masters use this plugin which means all of them can run any jobs that are sent to gearman server. We have lots of documentation on how we run the system in production.  

References

Versions

  • 0.2.0
    • Update for Jenkins 1.625.3 LTS and fix function registration.
    • When function changes, only register the delta instead of registering all functions for every node. This cut down the amount of CAN_DO update. New connection reset the state via RESET_ABILITIES to ensure a proper starting state.
  • 0.1.3
    • Send node labels back on build completion
  • 0.1.2
  • 0.1.1
    • Fix job result not being sent back to gearman client, check commit message for more info. 
  • 0.1.0
  • 0.0.7
    • Fix project-node registration. If a node matches any project label, register the generalized job and then also register it for each label in the intersection of project labels and node labels.
    • Supports Jenkins 1.502 to LTS 1.532.2
  • 0.0.6
  • 0.0.5
    • Set a node offline even if there is an exception
    • Always return WORK_COMPLETE when a build finishes regardless of the result
  • 0.0.4
    • Don't wait for the worker thread to join
    • remove restriction on slave to run single job at a time
    • Use more fine-grained synchronization in GearmanProxy
    • Rework starting/stopping executors
    • moved python examples to jenkins wiki
    • Add OFFLINE_NODE_WHEN_COMPLETE option
  • 0.0.3
    • ignore non-deterministic build failure and log it
  • 0.0.2
    • Bunch of fixes
    • ability to cancel gearman jobs from it's queue
    • ability to set jenkins job descriptions
  • 0.0.1 - initial release.

30 Comments

  1. I just tried this plugin, and but it seems not  share the slave.

    I don't understand what meanings that the gearman admin command "status"?   There are 3 columns in the result of "status" denotes what?

    1. I'm not sure what you mean by 'not share slave'.  Maybe you could provide a more detailed explanation?

      Here's a good description of the columns for the status info, http://preilly.me/2009/08/06/monitoring-gearman-over-telnet-port-4730

  2. I am a user of Jenkins gearman plugin and gearman-client

    assume this:

    1.jenkins job

    there is a Jenkins job: “JobTest”

     

    2.my expect

    my expect is that “JobTest” could not only run on many nodes(a node pool shares this job),but also we can specify which node it runs on.

    3.On the gearman client side:

    if we use build:function to submit the task,it runs on one of the node pool
    if we use build:function:nodename to submit the task,it runs on the specify node.

    How to reach this goal?

    i am looking forward to your help very much,thank you!

    1. build:function:nodename should execute the job on 'nodename'.  while build:function should run the job on any node.  The way to make your job runs on multiple nodes or executors is to make sure that your job is configured to 'Execute concurrent builds if necessary'.  That option is on the job configuration page.  Also when you use the simple python client do NOT set the '--wait' option.  

      Try running like so:

      python gear_client.py -s MyGearmanSever --function=build:myProject --jobs=10 --params='{..}'
      
      

      This tells gearman to execute 10 myProject jobs.  If your jenkins has 10 nodes or executors that can service the myProject it will run all of them concurrently.    

      1. very thanks but  sorry for my failure to express my question clearly.
        My goal:
        1、create a jenkins job
        2、use gear_client.py to submit task
        3、this job could not only share by many nodes but also runs on definite node (that mens:this job could run on one definite node if i specify the node,and the job could also run on one of the nodes pool)
        4、and a new question:if i have two jobs(jobB must run after jobB on the same node),they are shared by many nodes,how to submit this two tasks to make them run on the same node?

        what do i have to do to support these two goals?

        i am looking forward to your help very much,thank you!

        1. Your question doesn't sound any different.  I believe I have answered them in my previous post.  I think you may need to play around with the Jenkins job and node labels a little bit to understand how they work.  Once you have the labels setup to run your jobs against specific nodes then you can use the gear_client to execute the jobs on a specific node or on any node, but first you must setup the labels accordingly.

          to run 'myProject' on 'myNode'(set the job label to the corresponding node label) then run this:

          python gear_client.py -s MyGearmanSever --function=build:myProject:myNode --jobs=10 --params='
          
          

          to run 'myProject' on any node (leave the job label blank) then run this:

          python gear_client.py -s MyGearmanSever --function=build:myProject --jobs=10 --params='
          
          

          on question #4: again the same answer, first you need to setup your job and node labels to run jobs on the same node.  Then you can use the gear_client to direct which the jobs to a specific node or any node.

          1. assume this scene:

            our company use gearman clinet + jenkins + gearman plugin, 

            i create a job in jenkins as a service for us all, that mens every one of us  could use this service,

            userA want to sumbit this job to run on nodeA,

            userB want to submit this job to run on nodeB,

            userC want to submit this job to run on nodeC,

            that means they must modify the job's label every time to achieve this goal?

            1. No.  What you would do is assign your serviceJob to nodeA, nodeB and nodeC with project labels.  Set "Restrict where this project can be run: Label Expression" = nodeA || nodeB || nodeC

              When you start the gearman plugin the following functions should get registered:

              • build:serviceJob
              • build:serviceJob:nodeA
              • build:serviceJob:nodeB
              • build:serviceJob:nodeC

              if userA want to run serviceJob on nodeA then pass '-function=build:serviceJob:nodeA', for userB it will be '-function=build:serviceJob:nodeB', etc..

              but lets say userA want to run serviceJob but she doesn't care which node it runs on then she would pass '--function=build:serviceJob' and it will run on either nodeA, nodeB or nodeC, whichever node is available to run the job.

              1. thanks very much, you have solved my problem !

                by the way, if i create a jenkins job with no paratemer, 

                i use gearman client to submit this job

                the gearman plugin throw this exception:

                how to solve this problem, thanks.

                1. Probably a bug, could you please enter it into our issue tracker or better yet fix it :)

                  Anyways I think passing in no parameters is just the same as gear.Job('build:testjob:yangjun03') so a workaround is available.

                  1. gear.Job('build:testjob:yangjun03') dosn't work yet:


                    but gear.Job('build:testjob:yangjun03', '{}') works:


                     

  3. And another question, how to assign two different jobs running on the same node?

    1. This question is not specific to the gearman plugin.  In general you use the Jenkins job and node labels to assign jobs to nodes. 

  4. I tried the plugin,but it seems not share the slave well.

    I have a job1 on MasterA(not enough executors),it can't find MasterB to run job1(MasterB doesn't have job1 on it).I expect job1 can run on MasterB.

    How to reach this goal?

    i am looking forward to your help very much,thank you!

    1. You need to install the gearman plugin on MasterB and you also need to setup the same job on MasterB.  We duplicate our jenkins jobs across all of our masters with the help of the jenkins job builder application. 

      1. very thanks.I want job1 can run on MasterB'slave , and MasterB doesn't need have job1 on it . Is there any way to reach that goal? I know jenkins operation center can reach it ,but it's commercial.

        1. Unfortunately that's not possible with this plugin.  I think cloudbees has a commercial multi master solution that might be able to handle your use case.

  5. To approve the high availability of the gearman job server, I deploy two job servers on different machine. Can I configure plugin to register those job servers?

    I am looking forward to your answer, thank you!

    1. Unfortunately the jenkins gearman plugin currently only supports connecting to a single gearman job server. 

  6. I'm interested in a high availability solution without a single point of failure.

    It seem as though gearman isn't a good fit for this, because the gearman server becomes the single point of failure?

    1. Yes, that is correct.  The Gearman plugin is not a full HA solution.  It's a step better than just using a single master but there are definitely some missing pieces.  The two big requirements to get closer to HA are the ability to connect to multiple gearman job server and the ability to re-issue lost jobs.

      1. I've thought about using a JMS front end for this. At the least, it would hold onto jobs if the Gearman server is unavailable.

  7. I am using gearman (server,plugin,client) & jenkins job builder to create jobs . I have two jenkins master instances & major goal is to keep both of them in sync & share the load . I have few queries -

    1. Alternative to jenkins job builder , since it's tough for users to write yaml for every job they create . Any possiblity that we can use the jenkins UI to create the jobs which ultimately gets synced with the other jenkins instance .  

    2. When i send build request through gearman client to gearman server, only in this case it will use gearman workers to execute the build on both masters . If i use jenkins UI of one instance to build a job, it remains in queue & not shared with the other jenkins instance since i guess gearman server doesn't know about the request . 

    I understand the few things are little obvious but still i need the best way to deal with gearman . I repeat  I have two jenkins master instances & major goal is to keep both of them in sync & share the load . 

    Thanks in advance for your help !

    1. Yes, you can use the Jenkins UI to create jobs and those jobs will get registered with the gearman server (as long as the gearman-plugin has been started). After creating the jobs you must use a gearman client to schedule (or run) those jobs.  You can also run them by clicking the build button on the Jenkins UI but it will not get scheduled on the gearman queue which will not take advantage of multiple jenkins.    We use the scp plugin[1] to copy the build logs off of our jenkins server after every job run.   We've fixed up some things in that plugin so you'll definitely want to get the latest build of it which means you'll need to build it from source.

      [1] https://wiki.jenkins-ci.org/display/JENKINS/SCP+plugin

  8. Hi, I am using gearman in my infrastructure and recently we created a pipeline project using the workflow-aggregator plugin.

    Nothing worked and I checked the workers at gearman. None of my workers registered as able to build the pipeline project, does gearman cannot build pipeline projects?

    Thanks!

    1. As Khai Do does not answered yet, I found that gearman does not register workflow jobs because only jobs descending AbstractProject subtree are registered. The workflow-job descends the Job subtree.

      1. The initial gearman function registeration happen in ExectorWorkerThread.java[1].  When things change like new jobs, deleted jobs, new nodes, etc..  the listeners (ItemListener, SaveableListener and ComputerListener) will call the registerJobs[2] function to update the gearman function registrations.  You probably just need some additional code to handle the pipeline jobs in all of those locations.  Hope this helps.

        [1] https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/ExecutorWorkerThread.java#n120
        [2] https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/GearmanProxy.java#n339

  9. Seems gearman triggerred job lost the parameteres  which defined by the "This build is parameterized" function on the job, do you guys have any comments? Thanks.

    can't find any parameters defined in the job on the "Environment Variables" and "Parameters" list, although did add those parameters on job, and Global environment variables and ZUUL variables are not affected.

    Jenkins version : 1.625.3, zuul version: 2.5.2

    Oct 20, 2016 10:19:09 AM INFO hudson.plugins.gearman.MyGearmanWorkerImpl work
    
    ---- Worker 10.140.35.80_exec-2 executing function
    
    Oct 20, 2016 10:19:09 AM INFO hudson.plugins.gearman.StartJobWorker safeExecuteFunction
    
    ---- Worker 10.140.35.80_exec-2 scheduling test-jira-qualify-check build #10 on 10.140.35.80 with UUID 25cdb04a181b44e287eca0acfc7dac42 and build params [(TextParameterValue) BASE_LOG_PATH='43/215343/1', (TextParameterValue) ZUUL_PIPELINE='jira-qualify-check', (TextParameterValue) ZUUL_UUID='25cdb04a181b44e287eca0acfc7dac42', (TextParameterValue) LOG_PATH='43/215343/1/jira-qualify-check/test-jira-qualify-check/25cdb04', (TextParameterValue) ZUUL_VOTING='1', (TextParameterValue) ZUUL_CHANGE_IDS='215343,1', (TextParameterValue) ZUUL_PATCHSET='1', (TextParameterValue) ZUUL_BRANCH='master', (TextParameterValue) ZUUL_REF='refs/zuul/master/Z5d55f98e7c454289a2606412a16046b8', (TextParameterValue) ZUUL_COMMIT='3e22afb63c9e36e6bd67f944b894e4a0da7c7404', (TextParameterValue) ZUUL_URL='http://tcnbj-vm002.letv.local/p', (TextParameterValue) ZUUL_CHANGE='215343', (TextParameterValue) ZUUL_CHANGES='z/test/tom:master:refs/changes/43/215343/1', (TextParameterValue) ZUUL_PROJECT='z/test/tom']
    
    Oct 20, 2016 10:19:09 AM INFO org.gearman.common.GearmanJobServerSession submitTask
    
    Session GearmanJobServerSession:2001:GearmanNIOJobServerConnection:10.176.30.232/10.176.30.232:4730 is now handling the task 
    1. This is probably the same thing that was described in the 'known issues' section of this wiki.  Please check there to see if the workaround fixes this issue.