Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 22 Next »

This plugin uses Gearman to support multiple Jenkins masters. 

Plugin Information

View Gearman on the plugin site for more information.

Summary

We on Openstack infrastructor team use Jenkins extensively. Our jenkins servers, at peak load, runs 5000+ jobs per day.   At that load we require many jenkins slaves (200+) to process all of those build jobs.  We have found that our requirements were pushing Jenkins beyond it's limits therefore we've decided to create the Gearman Plugin to support multiple Jenkins masters.  The gearman plugin was designed to support extra slaves, allow load balancing of build jobs, and provide redundancy.  

Jenkins core does not support multiple masters.  You can setup multiple Jenkins masters but there is no coordination between them.  One problem with scheduling builds on Jenkins master (“MasterA”) server is that MasterA only knows about its connected slaves.  If all slaves on MasterA are busy then MasterA will just put the next scheduled build on the jenkin server queue.  Now MasterA needs to wait for an available slave to run the build.  This will be very in-efficient if your builds take a long time to run.  So.. what if there is another Jenkins master (“MasterB”) that has free slaves to service the next scheduled build on the server's queue?  Your probably saying.. “Then slaves on MasterB should run the build instead of waiting for slaves on MasterA to run them”, then I would say "good thought!".  However MasterB will never service the builds on MasterA's queue.  The client that schedules the builds must know about MasterB and then schedule builds on MasterB. This is what we mean by lack of coordination between masters. This  gearman-plugin attempts to fill the gap.

This plugin integrates Gearman with Jenkins and will make it so that any Jenkins slave on any Jenkins master can service a job in the queue.   This plugin will essentially replace the Jenkins (master) build queue with the Gearman job queue.  The job will stay in the Gearman queue until there is a Jenkins node (master or slave) that can run that job.  The gearman job queue is shared by multiple jenkins masters therefore the gearman can hand out jobs to the next available slave on any jenkins master.

Getting Started

This assumes some familiarity with Jenkins and Gearman

Install

  • Install the Gearman plugin like any other Jenkins plugin, refer to the Jenkins documentation.  You can also get the plugin directly from the Jenkins CI Repository
  • If you don't already have a Gearman server up and running somewhere you should install one.  This plugin will work with the following gearman servers:

Configuration

  

The Gearman plugin configuration should appear in the Jenkins global configuration page.  Click on the help bubbles if you need help with the configuration.  You should test the connection to your Gearman Server before running the workers.  The 'Enable Gearman' checkbox will start the gearman workers for the Jenkins server.  

Workflow

Starting the Gearman workers:

  1. When the gearman plugin is enabled a gearman worker thread is spawned for each executor on the master and slave nodes.  We'll call these "executor worker threads". Each executor worker thread is associated 1:1 with a Jenkins executor.
  2. We spawn one more thread to be a Gearman worker to handle job management.  We'll call it the "management worker thread" and it will register a "stop:$hostname" and "set_description:$hostname" Gearman function.  We use these functions to abort and rename jenkins jobs.  
  3. The gearman plugin will register gearman functions for each Gearman executor depending on the projects, labels and nodes that have been setup on the Jenkins master. You can check the registered gearman functions using the administration protocol.  It should look something like this..

Notes:

  1. Red text denotes gearman admin commands
  2. Blue text denotes gearman workers.  There is a default manager worker for the master and an executor worker for a jenkins executor on master.  There are two gearman executor workers for oneiric-668599 slave that maps to two jenkins executors on the same slave.  
  3. Functions like "build:guava:ubuntu" map to build:$projectName:$nodeLabel"

Here's the corresponding Jenkins:

Running a Jenkins build:

Any Gearman client (in any language) can connect to the Gearman server and send a request to build a Jenkins job.  The client just needs to provide the Gearman hostname, port, function, and UUID to start a jenkins build.  Here's an example in python..

import gear
import simplejson
import uuid
import time

#begin time
print time.asctime( time.localtime(time.time()) )

# Submit job
build_id = uuid.uuid4().hex
jenkins_build_params = {'param1':"cat",'param2':"dog",'param3':'bird','uuid':build_id}
job_name = 'build:guava'
client = gear.Client()
client.addServer('MyGearmanServer')
job = gear.Job(job_name, simplejson.dumps(jenkins_build_params), unique=build_id)
client.waitForServer()  # Wait for at least one server to be connected
client.submitJob(job)
print "Sending job "+job_name+" with uuid="+build_id+" and params= "+str(jenkins_build_params)

# wait for job to completefinished = False
while not finished:
    print "status = %s" % job.fraction_complete
    if (job.complete or job.failure or job.exception):
        finished = True
        print "completed = %s" % str(job.complete)
        print "failure = %s" % str(job.failure)
        print "exception = %s" % str(job.exception)
        print "warning = %s" % str(job.warning)
        print "data = %s" % str(job.data)
        print time.asctime( time.localtime(time.time()) )
        client.shutdown()
    time.sleep(1);
  1. Look in github for the Gearman StartJob client java examples.  
  2. The Gearman workers which were spawn by the Gearman Plugin will service any client request that come through to start and abort a Jenkins build.

Canceling a job:

  1. Any Gearman client can connect to the Gearman server and send a request to cancel a build.  Here's an example..
import gear
import simplejson
import uuid
import time

#begin time
print time.asctime( time.localtime(time.time()) )

# Submit job
build_id = uuid.uuid4().hex
jenkins_build_params = {'name':"guava",'number':"5"}
job_name = 'stop:MyGearmanServer'
client = gear.Client()
client.addServer('MyGearmanServer')
job = gear.Job(job_name, simplejson.dumps(jenkins_build_params), unique=build_id)
client.waitForServer()  # Wait for at least one server to be connected
client.submitJob(job)
print "Sending job "+job_name+" and params= "+str(jenkins_build_params)


finished = False
while not finished:
    print "status = %s" % job.fraction_complete
    if (job.complete or job.failure or job.exception):
        finished = True
        print "completed = %s" % str(job.complete)
        print "failure = %s" % str(job.failure)
        print "exception = %s" % str(job.exception)
        print "warning = %s" % str(job.warning)
        print "data = %s" % str(job.data)
        print time.asctime( time.localtime(time.time()) )
        client.shutdown()
    time.sleep(1);
ijhkhkjhkmport simplejson

Versions

  • 0.0.1 - initial release.
  • 0.0.3 - added features:
    1. ability to cancel gearman jobs from it's queue
    2. ability to set jenkins job descriptions

TODO

  • Dynamically update Gearman functions relative to an update to the Jenkins executors.
  • This pliugin does not register functions correctly for Jenkins Matrix Projects.

Plugin In Action

  • No labels