Child pages
  • Monitoring external jobs
Skip to end of metadata
Go to start of metadata

Plugin Information

View External Monitor Job Type on the plugin site for more information.

Adds the ability to monitor the result of externally executed jobs.

Jenkins is useful for monitoring the non-interactive execution of processes, such as cron jobs, procmail, inetd-launched processes. Often those tasks are completely unmonitored (which makes it hard for you to notice when things go wrong), or they send e-mails constantly regardless of the success or failure (which results into the same situation as you'll quickly start ignoring them anyway.) Using Jenkins enables you to monitor a large number of such tasks with little overhead.

Setting up a project

Create a new job and choose "Monitor an external job" as the job type.

Monitoring an execution

For Debian / Ubuntu:

sudo apt-get install jenkins-external-job-monitor
# obviously you should replace localhost with a FQDN if you want to run jobs from other machines.
export JENKINS_HOME=http://@localhost:8080/
java -jar /usr/share/jenkins/external-job-monitor/java/jenkins-core-*.jar "external-build-job-name" command-to-run
java -jar /usr/share/jenkins/external-job-monitor/java/jenkins-core-*.jar "external-build-test" ls -l

Once you set up a project, you can monitor an execution by running a command like this:

$ export JENKINS_HOME=http://user:pw@myserver.acme.org/path/to/jenkins/
$ java -jar /path/to/WEB-INF/lib/jenkins-core-*.jar "job name" <program arg1 arg2...>

For Windows:

> set JENKINS_HOME=http://user:pw@myserver.acme.org/path/to/jenkins/
> java -jar \path\to\WEB-INF\lib\jenkins-core-*.jar "job name" cmd.exe /c <program arg1 arg2...>

If your webserver extracts the jenkins.war file when it deploys Jenkins then you may use the path directly to the WEB-INF/lib directory and all other required jars will be found there. Otherwise you may extract these from the war file:

jenkins-core-*.jar
remoting-*.jar
ant-1.7.0.jar
commons-io-1.4jar
commons-lang-2.4.jar
jna-posix-*.jar
xstream-*.jar

All are found in the WEB-INF/lib path inside the war file. As long as they are all in the same directory, the java -jar /path/to/jenkins-core-*.jar command will find the other required jars.

  • Note: Older versions of Jenkins (before 1.324) also require winstone.jar in order to run this command. This jar is found at the top level directory inside the war file, and must be manually added to the classpath (with -classpath or -cp) in the java command.

The JENKINS_HOME variable is used to locate the server Jenkins is running, so this must be set. Unless your Jenkins job has build permission for guest users, include the username:password@ portion of the URL, as seen in the examples above.

  • Note: Authentication via username:password in JENKINS_HOME is added in Jenkins 1.324; with previous versions either grant anonymous build permission on the job, or use curl to post XML to Jenkins (see below).

You can copy jenkins-core-*.jar and the other required jars to other machines if you want to monitor jobs that are run on a different machine.

stdout and stderr of the program will be recorded, and a non-zero exit code will be considered as a failure.

Monitoring cron jobs

To monitor a cron job, simply run the above set up from your cron script. To avoid receiving e-mails from cron daemon, you might want to write something like:

JENKINS_HOME=http://myserver.acme.org/path/to/jenkins/
0 * * * *     export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-*.jar "backup" backup.sh 2>&1 > /dev/null

Note that you can also move the cron job itself to Jenkins by using free-style software project, which would also allow you to manually execute the job outside the scheduled executions.

Submit a run programatically

The above command submits the execution and its result by sending XML to HTTP. This means you can submit an execution record from any program, as long as you follow the same XML format.

The format is explained below:

<run>
  <log encoding="hexBinary">...hex binary encoded console output...</log>
  <result>... integer indicating the error code. 0 is success and everything else is failure</result>
  <duration>... milliseconds it took to execute this run ...</duration>
  <displayName>... The name to be displayed rather than the build number ...</displayName>
  <description>... Description of the build ...</description>
</run>

The duration element is optional, since 1.429 displayName and description can also optionally be appended, these three elements can be in any order, but must appear after log and result in that order. Console output is hexBinary encoded so that you can pass in any control characters that are otherwise disallowed in XML.

The above XML needs to be sent to http://myhost/jenkins/job/_jobName_/postBuildResult.

A simple example using curl would be (using no real data; the sequence 4142430A encodes a console output of ABC plus a trailing line feed symbol):

$ curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass@myhost/jenkins/job/_jobName_/postBuildResult

Submit a run per CLI

The easiest option is for the execution can be submitted per CLI/ssh command. The gzipped log file can be transported through pipe:

$ cat result.log.gz | ssh jenkins set-external-build-result --display 7d552c4ba_Linux_tb21 --job buildbot --result 1 -b --duration 42 --log -

Sometimes build number is needed, as the calling program might need to put an URL to the submitted build. The CLI command above returns the new build number. The command above can be called programatically from an Java application. You can find a sample for it here.

Changelog

Version 1.7 (Jan 06, 2017)

  • Allowing non-administrators to configure or delete jobs, given appropriate permissions.
  • Japanese localization update.

Version 1.6 (July 26, 2016)

Version 1.5 (July 12, 2016)

Version 1.4 (Nov 18, 2014)

  • Updated item label to match new noun-oriented convention.

Version 1.3 (Nov 14, 2014)

  • Refined where Build permission was checked to cover also the CLI command.

Version 1.2 (Aug 29, 2013)

  • new CLI command (pull #5)
  • JENKINS-19377 workaround for log rotation failure
  • NPE fix
  • translations

32 Comments

  1. Anonymous

    If you get:

    ----Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stdout copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
    Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stderr copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)

    ----call "-Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar" to your java call (modify the ant path!). So this may result in:
    export HUDSON_HOME=http://localhost:8080/hudson; java -Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar -jar hudson-core-1.136.jar "SVN_Backup" /opt/svn/svn_backup.sh

    - zeisss 

  2. How about running on a Windows machine. I've created a bat file like this:

    set HUDSON_HOME=http://localhost:8080
    java -jar C:\hudson_master\lib\hudson-core-1.200.jar "build_release_12.0" call_build.bat release_12.0

    I get this error (which indicates it cannot file hudsonhome): 

    Exception in thread "main" java.lang.NoClassDefFoundError: hudson/remoting/Callable
            at hudson.Main.getHudsonHome(Main.java:47)
            at hudson.Main.run(Main.java:37)
            at hudson.Main.main(Main.java:29)
    Caused by: java.lang.ClassNotFoundException: hudson.remoting.Callable
            at java.net.URLClassLoader$1.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClassInternal(Unknown Source)
            ... 3 more

     I'd think that this should work on a Windows machine as well.

    1. Unknown User (hudson@drewcox.com)

      I had this too (on Linux though), fixed it by copying the "remoting" jar out of the distro and into the same directory as the hudson-core jar.  The husdon-core jar has a long classpath defined in its manifest.mf, I haven't quite figured out what dependencies are required for remote monitoring, as I now have this error:

      Exception in thread "catalog.full.zip: stdout copier" java.lang.StringIndexOutOfBoundsException: String index out of range: -3
          at java.lang.String.charAt(String.java:687)
          at hudson.util.EncodingStream.write(EncodingStream.java:21)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:108)
          at hudson.util.DualOutputStream.write(DualOutputStream.java:29)
          at hudson.util.StreamCopyThread.run(StreamCopyThread.java:28)

  3. Hi I've been using Hudson for 2 months now and I must say that it is a really nice tools and there are so many available plugins.

    It is really nice.

    Today I started to play with the monitoring of external jobs and had an authentication error even when I set in the HUDSON_HOME env variable the userinfo user:password in the URL.

    Looking in the code Main.java , I added

     

                  String encoding = new sun.misc.BASE64Encoder().encode(new URL(location).getUserInfo().getBytes());
                  con.setRequestProperty ("Authorization", "Basic " + encoding);
    

    That works now but I am not sure this is perfect as it assumes that authentication is expected.

    Anyway that could perhaps be interesting to add such feature one way or another.

     Thanks

    Pascal

  4. Unknown User (oakley@bardo.clearlight.com)

    Is it possible to use this technique to send back data formatted as HTML? I'd like to be able to include some hyperlinks and bold text into the information that is generated on the console output page.

  5. Unknown User (josh@chapterthree.com)

    I am also wondering about log formats. Haven't had any luck (yet) getting intelligible results via HTTP POST and the XML format.

  6. Unknown User (prempola@gmail.com)

     Got this working using the following command.

    java -cp remoting-1.313.jar:hudson-core-1.313.jar:servlet.jar hudson.Main "Test" Test.groovy

  7. Unknown User (moon people)

    Example of working command:
    export HUDSON_HOME=http://localhost:8080/hudson
    java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh

    Details:
    I was having a lot of issues trying to get external jobs running successfully, but I figured it out. Here's what I did

    Hudson version: 1.315

    (running all this on OS X 10.5.6)

    1) created dir /opt/hudson
    2) installed apache-tomcat-6.0.20 into that directory (/opt/hudson/apache-tomcat-6.0.20)
    3) dropped hudson.war (version 1.315) into /opt/hudson/apache-tomcat-6.0.20/webapps dir
    4) started up tomcat (you may have a few config tweaks to make in server.xml withing your apache-tomcat-6.0.20 dir)
    5) went to http://localhost:8080/hudson saw that hudson was running ok
    6) Hudson -> New Job -> Monitor an external job (gave it a title of test-mon) -> OK
    7) cd into /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF
    8) set HUDSON_HOME=http://localhost:8080 (this can also be set to an entirely different hudson server. it does not have to be on your local machine to work.)
    9)from the /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF directory I ran the following:
    10) java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh
    11) Important: make sure to pass the language executor before the script (note: I ran 'ksh ./test_server_health.ksh' I tried many times without the 'ksh' and it doesn't work)
    12) went to http://localhost:8080/hudson and saw my test passed.

    the hardest thing I had to deal with was getting the classpath correct. you need remoting.jar, hudson-core-1.315.jar(of course the versioning will be different for you, 1.315 is the latest version as of today) & a servlet.jar (I chose servlet-api.jar from the apache lib directory out of frustration, so there may be another that exists that is the correct one.)

    in the command I provided I'm running a korn shell script (which is why you see the ksh before the actual script). you could run anything and typically groovy is sited as an example, but it can be bash, perl, ruby.

    also realized that where ever you set HUDSON_HOME to is where the test results get posted, so setting HUDSON_HOME=http://localhost:8080/hudson and looking for results on http://cool.company.com/hudson will make you go crazy

    good luck!!

  8. Unknown User (jpeck)

    It seems I also need jna-posix-1.0.3.jar (hudson 1.358)

  9. Unknown User (derekjchoi@yahoo.ca)

    I am attempting to connect a set of procmail executions and cronjobs with hudson, which works great.  However, one thing that I noticed during my testing is if the hudson server itself goes down, the executions don't run at all, which puts a dependency on Hudson CI being up.  Is it possible to pass  a command line argument to ignore errors communicating with hudson and continue with the execution?

    1. This may be too late for Derek, but perhaps it will help others.

      Here is a perl script that does the jenkins external job submission. The job will always run; Jenkins notification is secondary.

      https://gist.github.com/1237898

      1. Hi Andrew, I am interested in the perl script. The gist seems to have vanished. Would you mind putting it up again?

  10. 1. If you're getting java exeptions when setting up monitoring of external jobs that could be caused by the fact that you need commons-io-*.jar added to your client Jenkins setup. The list of 5 jars above seems incomplete.

    2. If your external job is memory-hungry (tries to grab more than 2Gb of ram) make sure you're running 64bit Java. Sounds obvious? Well, for me it wasn't ...

  11. Hi,

    I have setup up a jenkins system and trying to set up a job to monitor a cron job. My jenkins URL is http://dev1:8080 .This is what I'm doing on server dev02:

    JENKINS_HOME=http://dev01:8080

    export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-*.jar "Dev01 Report" kdev.sh 2>&1 > /dev/null

    I am getting below message

    Unable to access jarfile jenkins-core-*.jar

    I have created a monitoring job Dev01 Report on jenkins dashboard.

    I also tried

    export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-1.451.jar "SUN11 Report" kdev.sh 2>&1 > /dev/null

    but getting same error

    Unable to access jarfile jenkins-core-1.451.jar

    Can someone please help?

    Thanks.

    1. I managed to get past this one. I had to copy below files on to the remote server from $JENKINS_HOME/.jenkins/war/WEB-INF/lib/jenkins-core-*.jar
      remoting-*.jar
      ant-1.7.0.jar
      commons-lang-2.4.jar
      xstream-*.jar

      But now getting this error

      $ java -jar jenkins-core-1.451.jar "Dev02 Report" kdev.sh 2>&1 > /dev/null
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/io/input/NullInputStream
              at hudson.Main.remotePost(Main.java:151)
              at hudson.Main.run(Main.java:74)
              at hudson.Main.main(Main.java:56)
      Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.input.NullInputStream
              at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
              ... 3 more
      $

      Do I need to copy more files from $JENKINS_HOME/.jenkins/war/WEB-INF/lib to remote server?Any pointers anyone?

      1. You are missing the commons-io*.jar  In my release of Jenkins, its commons-io-1.4.jar.

        1. Tried running after copying commons-io*.jar file as well and got below error

          $ java -jar jenkins-core-1.451.jar "Dev02 Report" kdev.sh 2>&1 > /dev/null

          Exception in thread "main" java.lang.NoClassDefFoundError: org/jruby/ext/posix/POSIX
                  at hudson.Main.remotePost(Main.java:174)
                  at hudson.Main.run(Main.java:74)
                  at hudson.Main.main(Main.java:56)
          Caused by: java.lang.ClassNotFoundException: org.jruby.ext.posix.POSIX
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  ... 3 more

          Then copied jna-posix-1.0.3.jar and it went fine. So to summarize, I think we need below files on the remote server for remote monitoring to work properly. My server/os version is SunOS dev02 5.10 Generic_142909-17 sun4u sparc SUNW,Sun-Fire-V445

          Files copied

          jenkins-core-1.451.jar

          xstream-1.3.1-jenkins-9.jar

          ant-1.8.0.jar

          commons-lang-2.4.jar

          jenkins-core-1.451.jar

          remoting-2.12.jar

          xstream-1.3.1-jenkins-9.jar

          commons-io-1.4.jar

          jna-posix-1.0.3.jar

          Thanks for help.

          1. I have put together the steps I followed to "Monitor External Jobs" using Jenkins at link below:

            http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

            Hope everyone finds this useful. Feedback welcome.

  12. when i export JENKINS_HOME(which is not localhost path), i need pass username & password. but what can i do if the password exists character like '@'?

  13. when i export JENKINS_HOME(which is not localhost path), i need pass username & password. but what can i do if the password exists character like '@'?

  14. I'm using the latest rev of jenkins 1.472.

    I've set it up locally and tried both a remote and local test with ls -l

    This is what I see in the console ouput:

    Started
    FATAL: tried to access field hudson.model.Run.charset from class hudson.model.ExternalRun$2
    java.lang.IllegalAccessError: tried to access field hudson.model.Run.charset from class hudson.model.ExternalRun$2
    	at hudson.model.ExternalRun$2.run(ExternalRun.java:121)
    	at hudson.model.Run.execute(Run.java:1488)
    
    ...
    

    There is a ticket in the issue tracking system:

    https://issues.jenkins-ci.org/browse/JENKINS-14107

    If other people are experiencing this please create an account and vote for the issue.

    1. Hi,

      I have set this up to run both remotely and locally. I did face some issues initially but finally managed to fix it. I have put down the steps to setup jenkins from scratch at below link, can you please take a look and confirm that you haven't missed anything:

      http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

      I have set this up on solaris, but the steps should work on any *nix system. Please let me know if this helps.

      Thanks.

      1. Kumar,

        Thanks for our help.  Nope.  it didn't work.  I notice you are using version 1.451.    I tried it on solars and on my mac.  I get the same error.

        ...

        Update: Kumar, I rolled back to ver 1.464 and the ls -l example worked.  The latest release doesn't work

  15. Hi,

    Im enjoying every bit of jenkins so far, I need to send a notification email after a cron job finish, I see that there is no way to add post build tasks on this kind of project, I also tried the strawboss plugin but that doesnt work on newer jenkins versions (it seems outdated) is there a way to do this?

    I know a workaround is to use a ssh plugin, but I would like to know if there is something native in jenkins or with a plugin that only lets me add a post build step to these kind of jobs, thanks for the great work guys!

  16. So we have a number of Jenkins . . . and for the most part they are completely separate . . .

    However, we ran into the situation where one Jenkins didn't have enough slaves to run a particular assessment and so it go put onto a separate Jenkins.

    What I would really like is for the status of "that job on that Jenkins" to show up normally on "this Jenkins".

    Is that possible . . . this plugin seems to require a change to the way the extrnal process is involked . . . it really don't want to change the "remote" Jenkins job . . . only have it show up here too.

    Is that possible?

    Frank

  17. Hi,

    I can not find this external-monitor-job plugin in "available plugins" page. How can I install ?

    1. Hi,

      Why do you need such a plugin? I don't think one exists. What is it that you are exactly trying to do? If you are trying to monitor an external job then you need to configure this in jenkins. Follow the instructions listed at below link and let me know:

      http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

      Thanks,

      KVD

  18. How do i install this plugin, i don't see it in plugin manager -> Available plugins???

    1. As far as I can tell, it's now called "External Monitor Job Type Plugin".

  19. There seems to be a problem with the plugin and the jenkins CSRF protection.

    When I run my application with jenkins CSRF protection I get this error message:

    <html>
    <head>
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
    <title>Error 403 No valid crumb was included in the request</title>
    </head>
    <body><h2>HTTP ERROR 403</h2>
    <p>Problem accessing /job/test/postBuildResult. Reason:
    <pre> No valid crumb was included in the request</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>

    </body>
    </html>