Skip to end of metadata
Go to start of metadata
This plugin analyzes the causes of failed builds and presents the causes on the build page. It does this by using a knowledge base of build failure causes that is built up from scratch.
Saving statistics about failure causes is also possible.

Plugin Information

View Build Failure Analyzer on the plugin site for more information.

Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:

Knowledge base

The plugin comes with an empty knowledge base of failure causes. Populating this knowledge base is done by using the link "Failure Cause Management".
The link is shown only if the permission UpdateCauses is set for the current user. Press "Create new" and add a name and a description for the Failure Cause.
The description should contain the reason why this build failed as well as possible solutions for the build failure.

One or more optional categories can be added as a whitespace-separated list. These categories are only used for the statistics part of the plugin.
Add one or more Indications by pressing "Add Indication". By default, only one type of Indication exists, a "Build Log Indication".
Plugin developers can add new Indication types and more are planned to be added to the plugin itself.
The Build Log Indication searches through the build log, one line at a time, for a regular expression.

It uses Pattern.match, so the regular expression needs to match should be as follows

Match type


Single line

.*some words in the middle of a line.*

Multi line

some words here(.*\s)and some words here


Adding new failure causes and indications to the knowledge base.

From version 1.3.1 of the plugin, regular expressions can be tested on the Failure Cause Management page, in two different ways:

  • Writing a text in the text field shown above and testing against that.
  • Writing a URL to a build log in the text field. The plugin then runs through the log trying to match the regexp.

When accessing the Failure Cause Management page from a build, the URL will be added to the text field automatically.

The plugin comes with two ways of saving the knowledge base:

  • Local knowledge base. Saves the knowledge base in memory and serializes it as an xml file on the local Jenkins server (i.e. the "standard" Jenkins way of saving information).
  • MongoDB knowledge base. Saves the knowledge base in a Mongo database. This can be used to share the
    same knowledge base between servers. The knowledge base is still cached locally in-memory to avoid unnecessary database accesses.
    Plugin developers can add new knowledge base types.

Build log scanning

All builds on the server that are non-successful (aborted, failed, unstable) will be scanned for all failure causes. 
If an indication is found, the description will be put directly on the build page, with a link to the matching line in the build log.
If no cause is found, a text stating this will be shown. The text is configurable on the main configuration page of the server.

The build page when the build failure analyzer has found a failure cause.


The build log with the matching line marked in red.

Statistics

If MongoDB or some knowledge base type that supports statistics is used, statistics will be saved to that database. The same information
that is shown on the build page is saved to the database.

Administrative settings

On the configure system page in Jenkins, under Build failure analyzer, some new settings are available:

  • Enabled - since the plugin scans all failed builds, we felt the need to be able to disable the scanning. Uncheck to disable.
  • Text when no failure causes are found - text to show for failed builds whose failure cause is not found in the knowledge base.
  • Storage type - Jenkins Local or Mongo DB, as described above. For Mongo DB, configuration details to fill in are as follows: Host, Port, Database name, Username and Password for the Database, Enable Statistics logging (described above).
  • Convert knowledge base - If this check box is checked when the configuration is saved and a change has been made to the knowledge base settings, the data from the old knowledge base will be added to the new one. Note that duplicates could appear this way, so make sure that you untick this check box if you for example just have changed the username or password, or if you want to start with a clean knowledgebase.
  • Send notifications to Gerrit-Trigger-plugin - if enabled, will send the text for the found failure cause via the gerrit-trigger-plugin to Gerrit.
  • Concurrent scans - To speed up the scanning, each build will get a threadpool of this number of threads, with each thread handling one indication. For a small system, 3 is usually enough.

Token Macro integration

Usage:

${BUILD_FAILURE_ANALYZER, includeTitle=true, includeIndications=true, useHtmlFormat=true, noFailureText="Sometext"}

Parameters (=default):

  • includeTitle (=true) -- When true, the "Identified problems:" title will appear over the causes.
  • includeIndications (=true) -- When true, the indication numbers and links into the console log are included in the token replacement text.
  • useHtmlFormat (=false) -- When true, the replacement will be an HTML snippet.
  • wrapWidth (=0) -- Wrap long lines at this width. If wrapWidth is 0, the text isn't wrapped. Only applies if useHtmlFormat == false.
  • noFailureText (="") -- Text to provide if there are no found failures.

Placeholders in description

Replace description placeholders from captured expressions in found indications

Substitutions may be made within the description with placeholders of the form ${I,G}, where I is the indication number and G is the captured group within the indication expression. e.g., ${1,1} would be replaced with the first indication's first captured group and ${1,2} would be replaced with the first indication's second captured group.

Tips & Tricks

Aggregate statistics to Graphite

If you are using the MongoDB KnowledgeBase, you can use these scripts in a cron job to aggregate the statistics into Graphite. Note that they only work on mongo 2.x node js driver

Known bugs

A full list of open bugs raised against build-failure-analyzer component..

Key Summary T Created Updated Assignee Reporter P Status
Loading...
Refresh

Change Log

Version 1.20.0 (released Jun 14, 2018)

  • Moved to a slightly more modern minimum Jenkins core version (2.7.3) (Pull #81, #83)
  • Display HTML for cause description in matrix (Pull #84)
  • Improve support sorting "Never" on cause management page (Pull #86)

Version 1.19.1 (released Jan 8, 2018)

Version 1.19.0 (released May 5, 2017)

  • Use fixed pool size, parse all single line matchers in one thread (Pull #57)

Version 1.18.1 (released Mar 7, 2017)

  • Corrected JSON structure of RabbitMQ message (Pull #66)

Version 1.18.0 (released Feb 17, 2017)

  • Added a FailureCauseProvider for the MQ Notifier plugin (Pull #64)
  • JENKINS-41279 Fixed broken div wrapping (Pull #63)
  • Better feedback to Gerrit when no FailureCauses are found for a build (Pull #61)

Version 1.17.2 (released Oct 21, 2016)

  • JSON Serialization fix for MultiLineBuildLogIndication (Pull #59)
  • Performance fix for MultiLineBuildLogIndication (Pull #59)
  • Line break fix (Pull #58)

Version 1.17.1 (released Sep 5, 2016)

  • Fix Gerrit feedback for nested builds (Pull #56)
  • Introduced and separated out Renderer for report formatting (Pull #55)
  • Escape single quote correctly (PR #54)

Version 1.17.0 (released Aug 17, 2016)

  • Add configurable limit for log size to be scanned. (Pull #53)

Version 1.16.0 (released Jun 20, 2016)

Version 1.15.0 (released Apr 19, 2016)

Version 1.14.0 (released Apr 5, 2016)

Version 1.13.5 (released Feb 17, 2016)

  • Added build.displayName to stored statistics. (Pull #42)

Version 1.13.4 (released Feb 5, 2016)

  • Some minor formatting fixes in UI strings. (Pull #40)
  • Scanning-threads now gets the name of the indication and pattern currently being scanned for, for easier thread dump analysis. (Pull #41)

Version 1.13.3 (released Jan 5, 2016)

Version 1.13.2 (released Nov 25, 2015)

  • Fixed a issue when testing expressions on build logs located in folders. (Pull #36)

Version 1.13.1 (released Sept 25, 2015)

Version 1.13.0 (released Apr 10, 2015)

Version 1.12.1 (released Jan 16, 2015)

Version 1.12.0 (released Jan 15, 2015)

  • JENKINS-24434 fix trim() usage. (Pull #29)
  • Configurable anonymous access to the list of failure causes. (Pull #31)
  • Having the token expand into some text when no failure cause is identified. (Pull #32)

Version 1.11.0 (released Nov 27, 2014)

  • Failed Tests can be shown as failure causes, but not counted in the statistics (Pull #25)
  • [UI] Added space between indication links. (Pull #26)

Version 1.10.3 (released Oct 13, 2014)

  • One more fix for icons not correctly displayed (Pull #24)

Version 1.10.2 (released Oct 7, 2014)

Version 1.10.1 (released Sep 30, 2014)

Version 1.10.0 (released Sep 19, 2014)

Version 1.9.1 (released Jun 25, 2014)

  • "Failure Scan Options" is not hidden for users without build or configure permissions, it also hides when scanning is turned off.

Version 1.9.0 (released Jun 18, 2014)

  • Statistics upstream link - added upstream link info to statistics.
  • JENKINS-18518 Red highlights not showing when clicking on indications (pull #21)
  • BuildFlow Dependencies And Nested Failure Causes in Gerrit (pull #18)
  • Address multiline "Match Text" failure (pull #17)

Version 1.8.1 (released May 22, 2014)

  • Address NPE in getNotScannedBuilds() (pull #16)

Version 1.8.0 (released May 19, 2014)

  • Project page shows last build failure cause
  • Failure Cause Management link hidden for projects with disabled scanning

Version 1.7.0 (released Apr 1, 2014)

  • Multi line build log indications
  • Optionally store statistics about successful builds

Version 1.6.0 (released Mar 10, 2014)

  • Ability to re-scan non scanned builds (for new installations) and all builds for a project
  • Graphs on projects, slaves and master(s) if using a statistics logging enabled knowledge base (like the MongoDB Knowledge base)
  • Shows failure causes from downstream builds directly on the upstream build page.
  • ListView column showing the failure cause of the last build, if there is one.

Version 1.5.1 (released Nov 19, 2013)

  • Fixed an XSS vulnerability

Version 1.5.0 (released Apr 24, 2013)

New Features
  • The found failure cause is exposed to the REST Api (jobX/1/api).

Version 1.4.1 (released Mar 14, 2013)

Bugs fixed

Version 1.4.0 (released Feb 15, 2013)

New Features
  • Possibility to test regexps on a build log
Bugs fixed
  • Log annotation bugfixes.
  • JENKINS-15948 Build Failure Analyzer icons aren't displayed if Jenkins isn't installed at root context.(again)
  • JENKINS-15926 Build Failure Analyzer with Timestamper output ugly.(again)
  • JENKINS-16596 Repeat/double loggin issue due to Build failure Analyzer.
  • JENKINS-16104 Build Failure Analyzer: Ugly output from plugin.
  • NPE fix when a slave is taken offline during a build.
  • Fix for internal serialization of matrix aggregated indications.
  • Small UI fix in failure cause management page.

Version 1.3.0 (released Dec 06, 2012)

New Features
  • Possibility to test regexp on a line of text when editing BuildLogIndications
  • Output from Build Failure Analyzer shown in normal console.
Bugs fixed
  • JENKINS-15986 Cannot save job configuration pages on Jenkins 1.463 or newer.
  • JENKINS-15948 Build Failure Analyzer icons aren't displayed if Jenkins isn't installed at root context.
  • JENKINS-15926 Build Failure Analyzer with Timestamper output ugly.
Other
  • Updated Gerrit Trigger optional dependency: 2.7.0

Version 1.2.0 (released Nov 22, 2012)

Initial Release

Known issues

Breaks Job configuration pages on Jenkins 1.463 or newer
The problem we have seen is when loading configuration pages. We don't have a registered descriptor for our notifier,
hence, when the hetero-list for the post build actions is generated, an Exception is thrown.

47 Comments

  1. Exactly how is the "Job configuration pages" broken?  I have been getting some odd behavior on my config pages after installing - I can't typically save or apply until I hit a "delete" button down in an unlabelled section of the configuration.  Is there a JIRA incident open that I can watch for a fix?

    1. I opened JENKINS-15986 for the job configuration page problem.

  2. This plugin will probably be what I need to switch over from Cruise Control.NET because I can easily use regex to search in the logs for any kind of information with xsl.

    However, can the output of this plugin be used in the email too? In Cruise Control.NET, what is present in the email is identical to the build result dashboard. Thanks.

    1. We could probably make a solution by integrating our plugin with the email-ext plugin, we will add it to our backlog :)

      1. Hi, I do vote for this too because email messages are very popular and especialy for the known troubles could help the users to see the problem quickly.

        Thanks.

        zakyn

  3. This is a very useful plugin, but now it has some seriuos bugs:

    1. When a problem is identified, the build log is duplicated - it becomes two times large, but the second part is the same log + junk text, for example:  [8mha:AAAAdB+LCAAAAAAAAABb85aBtbiIQSOjNKU4P0+vIKc0PTOvWK8kMze1uCQxtyC1SC8ExvbLL0llgABGJgZGLwaB3MycnMzi4My85FTXgvzkjIoiBimoScn5ecX5Oal6zhAaVS9DRQGQtr4jMKkbAI50d5WBAAAA[0m exec 27.12.2012 14:12:51 liquibase.database.sql.CreateTableStatement getSqlStatement

    2. In many cases the highlighted line in the build log is not the right line, but the line after the correct one or 1-2 lines lower.

    3. When a problem is identified and I click on Indication link, it is just build log open, but not the place where the indication is found.

    All these problems seem to take place when the build log is not small. I have this problem with logs of 1.5 Mbytes and more.

    1. I wonder if this is perhaps an interaction with mixed end-of-line styles in your log?  If you find a few mixed EOLs, that would be a clue for the developer.

    2. These problems have all been taken care of in the 1.4.0 release.

  4. I'd love it if the failure explanation were visible other than by clicking a tiny lightbulb icon.  At least hovering over the icon should pop it up.  But also, the explanation should appear in the job page.

    1. The explanation does already appear on the build page (rather than the project page).

  5. Updated to 1.4 but still don't get the icons.

    The link comes out as http://127.0.0.1:8080/jenkins//plugin/build-failure-analyzer/images/16x16/information.png

    It would work if it was:

    /jenkins/plugin/build-failure-analyzer/images/16x16/information.png

    Otherwise, this is a great plugin. Thanks.

    1. The double slashes in the beginning does no harm for us, not sure what is different for you.

      Anyway, this should be a quite easy bug to fix, we just need to make sure we aren't adding double slashes anywhere.

  6. I just found this plugin and like it already despite the drawbacks discussed above.  I do TONS of customer support on builds and this could help a lot.

    Feature requests:

    1. Would like to have the definitions in source control and shareable across multiple Jenkins instances.  I don't want to have 20 separate databases for 20 different Jenkins.  That is not to say you need to have some kind of "share mode" that allows updates on a file server, but honor the read-only bit and don't attempt/allow updates unless it is clear.  Also keep all the definitions in a single file or directory so they can be easily isolated and moved from Jenkins to Jenkins.  (Haven't looked yet how your are currently doing this yet).

    2. Would like to have "and" as well as "or" in the definitions . . . In my case, a Python Exception can come from several different sub-systems and after finding "Traceback" I would like to be able to explore the Traceback and assign blame based on source path.  So it would be nice if the "or" search could be limited to the "first line plus a few".

    I have done a "poor man's" version of exactly this using the "Log Parser Plugin", but this is much more expressive, allows more categories and (the big one for me) an explanation of what the user should do to fix the issue.

    This could save me a ton of customers calls . . . as time allows, I would be happy to help in expanding this plugin with code or docs if you are interested.

    Frank

    1. Sorry for the late reply. Glad that you like the plugin :)

      1. We currently have 2 ways of saving the failure causes; either locally where you get the issue you express, or using MongoDB.

      If you use MongoDB you can point all of your 20 Jenkins instances to the same database.

      You could also implement a new way of saving failure causes and do a pull request :)

      2. This is a commonly requested feature and something we would like to do. The feature was discussed internally before the first version of the plugin was created.

      We haven't had time yet to look at a solution for this but would gladly accept contributions.

      Br Tomas

  7. Regarding the icon bug om Build History, I think the problem is in PluginImpl at line 256.

     return Hudson.getInstance().getRootUrl() + contextPath + getImageUrl(size, name);
    

    Entering the following code into the Jenkins Script console

     Hudson.getInstance().getRootUrl()
    

    Shows that the context path is already appended to Hudson.getInstance().getRootUrl()

  8. Is there any way to force a rescan of all failed builds? The doc reads like the log is scanned only when the build fails. 

    Kean

  9. sorry, I thought I had a solution for this, but it does not work yet

  10. It is mentioned that you can access the failure causes through the REST API.  I am not seeing this at all, what is the expected output?

    1. Actions doesn't show up on the default depth, you need to go a bit deeper, but I think you only need one extra depth IIRC.

  11. I would be perfect if this plugin was able to take a node offline in case of a specific failure.

    There are other plugins to do that, but none of them are as flexible as this one.

    1. We have planned for a feature where the plugin could perform actions when a certain failure cause is detected such as take a node offline, send a mail to admin etc. It should also be extendable so that other plugins can provide actions as well.

      1. Are there any plans to implement this feature?

        1. Not that I know of.

          Contributions are always welcome (wink)

  12. Hi is there any plan to add Email notifcation or some way of integration with Email -Ext ?

    I have tried use a customer template but looks like BFA action scans happening after Email is being sent so there is no values to put into email.

    However when testing a Email Template (using plguin test)  after build it works fine.

  13. If the failure causes are stored in the Mongo database, it's not straightforward to produce a dump of the data. But this can be done through the Script Console (under Manage Jenkins):

    	import com.sonyericsson.jenkins.plugins.bfa.*
    	
    	def failureData = new groovy.json.JsonBuilder();
    	def failureCauses = [];
    	
    	for (cause in PluginImpl.getInstance().getKnowledgeBase().getCauses()) {
    	  def failureCause = {};
    	  def indications = [];
    	  
    	  for (indication in cause.getIndications()) {
    	    indications.add(indication.toString());
    	  }
    	
    	  failureCauses.add([id: cause.getId(), name: cause.getDisplayName(), categories: cause.getCategories(), indications: indications]);
    	}
    	
    	failureData{"causes" failureCauses};
    	
    	print failureData.toPrettyString();
    
  14. To re-run the Build Failure Analysis for a single build (Script Console under Manage Jenkins):

    def jobname = "name_of_your_job";
    def buildnumber = 1234;
    
    // Find the job and build objects
    def job = hudson.model.Hudson.instance.getItem(jobname);
    def build = job.getBuildByNumber(buildnumber);
    
    // Remove the existing BFA analysis (this removes one instance at a time)
    def BFA = build.actions.find{ it instanceof com.sonyericsson.jenkins.plugins.bfa.model.FailureCauseBuildAction };
    build.actions.remove(BFA);
    
    // Re-run the BFA analysis on the build
    def fos = new FileOutputStream(build.getLogFile(), true);
    def log = new PrintStream(fos, true, "UTF8")
    com.sonyericsson.jenkins.plugins.bfa.BuildFailureScanner.scan(build, log);
    
    // Save the build
    build.save();
    
    1. I just tried to use the re-run groovy script, but get the following error: groovy.lang.MissingPropertyException: No such property: hudson for class: Script1

      Any idea how to fix it?

      1. I ended up using this:

        def job = jenkins.model.Jenkins.instance.getItemByFullName(jobname); 

        instead of

        def job = hudson.model.Hudson.instance.getItem(jobname);

  15. Hi all,

    I'm trying to get the multiline indications working for this plugin but I'm having issues. I wrote regex that works in regex testers but when I paste it into the plugin it doesn't work. 

    When I go back in to edit the indication the plugin has appended and prepended regex to mine. 

    Has anyone got a sample of multiline regex that works in this plugin?

    Sean

  16. Great plugin!!

    Just two minor improvements

    1.- Include a title for the extra column showing the failure cause in views

    2.- Show ALL the failure causes in this extra column.

  17. Hi,

    Is there any possibility to update the knowledge base by script?

    I see there are unique values in the build-failure-analyzer.xml, which probably generated when adding entry from the GUI.

    I have causes list which is changing from time to time.

    It will be great if we could monitor those changes by script.

    Thanks!

  18. I was using the plugin very successfully with the local files.  I wanted to get better statistics, so I setup a Mongo DB and tried to convert, which is causing some problems.

    Could not fetch causes: Read operation to server <MONGODB SERVER> failed on database build_failures

    Do I need to create the database before configuring in Jenkins, or does the plugin automatically create it?

    If so, what should I populate it with?

    I don't see anything in the MongoDB logs.

  19. 1) Great plugin

    2) Is there a possibility to scan also successful builds? (I know that it is not consistent with the name of the plugin, but still.) Or is there another plugin for displaying something in the same style from the console output on the job's main page?

  20. Hi, I really like this plugin but it is currently not working for us. It has started to log messages along lines...
    Jan 21, 2016 2:27:59 PM WARNING com.sonyericsson.jenkins.plugins.bfa.model.FailureReader scanMultiLineOneFile
    File timeout scanning for indication '.make .*Shell. failed.*' for file log
    Any ideas why I am starting to get timeouts?

    The actual timeout limits are hard-coded in FailureReader. (e.g. private static final long TIMEOUT_FILE = 10000;)

    1. Sorry formatting went bit funny...here's a better sample log

      Jan 21, 2016 2:28:08 PM WARNING com.sonyericsson.jenkins.plugins.bfa.model.FailureReader scanMultiLineOneFile
      File timeout scanning for indication '.*The process cannot access the file because it is being used by another process.*' for file log

      I'm using Storage type local and only have 7 causes and 30 jobs. Only other recent change is adding more slaves.

  21. Can this plugin be used to automatically fail a build?  It doesn't look like it although it would be perfect for that.  I just installed https://wiki.jenkins-ci.org/display/JENKINS/Log+Parser+Plugin to do that although the user interface for this plugin would be better, e.g., edit the patterns inside jenkins instead of an external file.

  22. Does anyone have any examples of using this with the job-dsl-plugin?  I see support was added but can't find how to use it.

  23. Hi, it would be really useful if possible to set the build to failed when some cause pattern is matched to the build log.

    It was really sad to realize that "rescan all" only checked the failed builds, leaving the "Successful" builds as is even though they have the very same string in the build log...

  24. Hi and thanks for this fantastic plugin.

    We have in our configuration a "master build" matrix job which is triggered either manually, or from CI SCM hooks via parameters. Is there any way for the Build Failure Analyzer plugin to scan the *invoked sub-project* in addition to the log from the current job? We have hooked the CI jobs to physical TVs that display build status and "action points" for solving issues for every project. It'd be very convenient if they could show the action points in the invoking project, instead of forcing developers to go to Jenkins and navigate to the failed sub-project to see the failure cause(s). The plugin could either traverse sub-projects and scan them as well, or take an argument of additional logs to scan (where we'd just hook it up to the logs of the sub projects).

    Thanks for considering this,

    - Tommi

  25. Hello!, I have the same problem with icons not being displayed when I run Jenkins from another address instead of localhost, since the link at the icon is always based on the localhost address

  26. I don't know if this is user error or a bug:

     

    I added a "compilation failures" category, with a Build Log indication of:  ^.*? cannot find symbol

    I have a job where this should have matched several times.  On the build summary page, I see my category and description, and then a singular link for "Indication 1".   The link takes me to the log file where only one spot is marked.   On the build page, there's a down arrow next to the "Indication 1" link (when hovering), however it doesn't drop down any options.  I'm thinking that was supposed to present a list of all occurrences of that particular indication.  Is this correct, but just not implemented?  As well, when looking at the build.xml on the server, there is only one FoundIndication in the indications list, rather than the expected 2 (in my case).

  27. Connection to mongo3.4 with enabled authentication fails. 

  28. Is there any way to support adding basic environment variable replacement in the description field?  Specifically, $BUILD_URL in order to link directly to a build report, but could be useful for others as well.  For example, in the case of a Checkstyle failure, we could add a link to "$BUILD_URL/checkstyleResult/HIGH/"

  29. Hi!

    Is it possible to check any text file for "errors" not just build log? It would be wonderful.

  30. Hello,

    I have successfully configured the BFA and use it in a Jenkins declarative pipeline.
    Now, I would like to add the BFA result within a mail notification when it fails.
    In other words, I am looking for a way to display ${BUILD_FAILURE_ANALYZER, includeTitle=true, includeIndications=true, useHtmlFormat=true, noFailureText="Sometext"} within mail.

    Thanks for your help.


    Best Regards
    JiElPe

    1. We use it in our email without any problems.   Just place it in the body and it's good to go.   Is it not working for you?

      1. Hi Adam,


        Well, it does not work.
        I have the following mail part for failure :


                   emailext(
                        to:getUserMail() + ",${JiotbMailAdmin}",
                        mimeType: 'text/html',
                        subject: "${JOB_BASE_NAME} is ${currentBuild.currentResult} on ${params.branchName}",
                        body: """
                            <p>Status Set to ${currentBuild.currentResult} on ${params.branchName}</p>
                            <p>See Build ${env.JOB_BASE_NAME}#${env.BUILD_NUMBER} <a href=\"${env.BUILD_URL}/console\">console log</a>.</p></h4>
                            <p>${env.BUILD_FAILURE_ANALYZER}</p>
        				"""

                The mail is sent with all awaited values, unless ${env.BUILD_FAILURE_ANALYZER} which is null though the BFA has actually found something.