Child pages
  • JENKINS-41661 Post-Mortem
Skip to end of metadata
Go to start of metadata

On February 2nd, CloudBees released several plugins which required a synchronized release. To achieve this synchronization, we preemptively blocked those releases from the public update center. 

Unfortunately, there were typos in the premeptive block which allowed the github-branch-source 2.0.1 and cloudbees-bitbucket-branch-source 2.0.2 and  release to still be visible in the update center. We didn't realize this until more than 18 hours after the release.

Bug

JENKINS-41661

Severity

Severe - Unable to use Jenkins

Executive Summary

GitHub Branch Source 2.0.1 and BitBucket Source 2.0.2 were available in the update center, but dependencies were not. As a result, Jenkins would not completely start.

User Impact

25 watches on the ticket. Jenkins does not fully load.

How did we find out?

User reports

Incident Type

Regression

Mitigation efforts

We removed the plugin from the update center and proactively scrubbed the metadata from mirrors. We also provided a zip file of all the needed dependencies on the ticket.

How did this bug escape?

We failed to test the update center after releasing the plugins. However, the Jenkins update center does not protect against this problem, but has enough data to do so.

So while we should have tested this, Jenkins should do a better job of protecting users from this situation.

Actions to prevent this 
from happening in the future

If we put an update center block, we should test it once the releases are available. However, this generally seems to be a fragile approach and I think it should be avoided.

CloudBees should instead use a staging server like Nexus to perform coordinated releases (when necessary) so that they can be pushed to the Jenkins infrastructure as a whole.

Finally, we should fix the Jenkins update center so that it doesn't screw the user if the update center is broken like this.(JENKINS-23757)

  • No labels