Child pages
  • Jenkins says my reverse proxy setup is broken
Skip to end of metadata
Go to start of metadata

Since Jenkins 1.572 this message can also appear if you don't access Jenkins through a reverse proxy: Make sure the Jenkins URL configured in the System Configuration matches the URL you're using to access Jenkins.

Symptoms

An error message is displayed in the "Manage Jenkins" page - "It appears that your reverse proxy set up is broken"

Background

For a reverse proxy to work correctly, it needs to rewrite both the request and the response. Request rewriting involves receiving an inbound HTTP call and then making a forwarding request to Jenkins (sometimes with some HTTP headers modified, sometimes not). Failing to configure the request rewriting is easy to catch, because you just won't see any pages at all.

But correct reverse proxying also involves one of two options, EITHER

  • rewriting the response (for more information see Hyperlinks in HTML). The primary place where this needs to happen is the "Location" header in the response, which is used during redirects. Jenkins will send back "Location: http://actual.server:8080/jenkins/foobar" and the reverse proxy needs to rewrite this to "Location: http://nice.name/jenkins/foobar". Unfortunately, failing to configure this correctly is harder to catch; OR
  • Setting the X-Forwarded-Host (and perhaps X-Forwarded-Port) header on the forwarded request. Jenkins will parse those headers and generate all the redirects and other links on the basis of those headers. Depending on your reverse proxy it may be easier to set X-Forwarded-Host and X-Forwarded-Port to the hostname and port in the original Host header respectively or it may be easier to just pass the original Host header through as  X-Forwarded-Host and delete the X-Forwarded-Port header from the request. You will also need to set the X-Forwarded-Proto header if your reverse proxy is changing from https to http or vice-versa

So Jenkins has a proactive monitoring to make sure this is configured correctly. It uses XmlHttpRequest to request a specific URL in Jenkins (via relative path, so this will always get through provided the request is properly rewritten), which will then redirect the user to another page in Jenkins (this only works correctly if you configured the response rewriting correctly), which then returns 200.

This error message indicates that this test is failing - and the most likely cause is that the response rewriting is misconfigured. See the Server Configuration Guides (below) for additional tips about configuring a reverse proxy. 

Note. The reverse proxy tests were improved in release 1.552 so users with previously working proxy setups may start to receive proxy warnings. 

Be sure to set the X-Forwarded-Proto header if your reverse proxy is accessed via HTTPS and then Jenkins itself is accessed via HTTP i.e. proxying HTTPS to HTTP.

Changing the context path of Jenkins with a reverse proxy is fraught with danger. There are lots of URLs that you need to rewrite correctly, and even if you get the ones in HTML files you may miss some in javascript, CSS or XML resources.

The recommendation is to ensure that Jenkins is running at the context path that your reverse proxy is serving Jenkins at. You will have the least pain if you keep to this principle.

While it is technically possible to use rewrite rules to change the context path, you should be aware that it would be a lot of work to find and fix everything in your rewrite rules and the reverse proxy will spend most of its time rewriting responses from Jenkins. Much easier to change Jenkins to run at the context path your reverse proxy is expecting, e.g. if your reverse proxy is forwarding requests at https://manchu.example.org/foobar/ to Jenkins then you could just use java -jar jenkins.war --prefix /foobar to start jenkins using /foobar as the context path

 

Further Diagnosis

For further diagnosis, try using cURL:

curl -iL -e http://your.reverse.proxy/jenkins/manage \
   http://your.reverse.proxy/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

(assuming your Jenkins is located at http://your.reverse.proxy/jenkins/ - and is open to anonymous read access)

Server Configuration Guides

While the pages talk primarily about Apache / NGinX / HAProxy / Squid, they also have information that applies to other reverse proxies.

If using Apache check that nocanon is set on ProxyPass and that AllowEncodedSlashes is set as per the Apache link above.

AllowEncodedSlashes is not inherited in Apache configs, so this directive must be placed inside the VirtualHost definition.

  • No labels

28 Comments

  1. If you're having this issue with Jenkins behind nginx, you can fix it with the following location block:

      location ^~ /jenkins {
        proxy_pass          http://localhost:8080;
        proxy_read_timeout  90;
     
        # Fix the “It appears that your reverse proxy set up is broken" error.
        proxy_redirect      http://localhost:8080 $scheme://example.com;
     
        # Optionally, require HTTP basic auth.
        # auth_basic "Please authenticate to use Jenkins";
        # auth_basic_user_file /opt/nginx/htpasswd;
      }
    

    This is taken from my full post on Jenkins and nginx at http://www.phase2technology.com/blog/running-jenkins-behind-nginx/.

    1. Replace "your.reverse.proxy" with the address of your reverse proxy. =)

    1. The expected result is a 200 OK, and the URL is correct. Most likely your proxy processes the encoded slash in the second URL and sends .../a/b/ instead of .../a%2Fb/ to Jenkins.

      If you're using nginx, proxy_pass is the most likely culprit here. Using anything other than scheme://ip:port; or scheme://backend; causes nginx to process the URI before transferring it. For example http://127.0.0.1:8080/jenkins; or even just http://127.0.0.1:8080/; already cause it to be processed, whereas http://127.0.0.1:8080; like in the example above does not.

      If you're using Apache and mod_proxy, your ProxyPass should include nocanon, I believe.

      1. At http://stackoverflow.com/questions/20496963/avoid-nginx-decoding-query-parameters-on-proxy-pass-equivalent-to-allowencodeds, a work around should to use

        location / {
           proxy_pass  http://tracking/webapp$request_uri;
        }
        

        but Jenkins redirects on itself

  2. After upgrading to 1.554 jenkins started to complain about the reverse proxy setup. (I don't know what the previous version was).

    Given the fact that we use mod_jk and not a HTTP proxy I don't see how I can fix this problem. As far as I can tell it's working ok.

    Also, the reverse proxy test doesn't work for me either, it gives a 404. A 404 from the proxy, and not jenkins.

    1. It is the same for me. I use mod_jk since we migrated from Hudson to Jenkins. Ever since, I use mod_jk with https.

      After we upgraded to 1.574 we are stuck with this error message. The AllowEncodedSlashes NoDecode option was already set.

      Can someone please help?

  3. Yes happened to me too after upgrading to 1.554. Worked perfectly before.

  4. The check in 1.552 is stricter, so it will report a warning for setups that earlier versions would consider working.

  5. I got the message about reverse proxy and some investigation showed that our DNS server did not answer to the name, I had put into the field "Jenkins URL". It was a DNS error but it could also happen in case of a typing mistake. If you want to test if that's the case on your Jenkins server, you can replace the server name with the IP number e.g. http://192.0.2.2:8080/ instead of http://example.server.invalid:8080/ - the IP number version did complain.

  6. You will also get this error if you are not using apache but start your jenkins with the --prefix option

  7. We are seeing this warning since upgrading from Jenkins 1.570 to 1.579.  We use the 'pound' reverse proxy server and our configuration for Jenkins is:

    # Jenkins

       Service

       URL "^/jenkins/.*"

          BackEnd

             Address [snip].com

             Port 8090

          End

       End

    Any suggestions how to fix this please?

  8. For Apache, if your virtual hostname is not the same as your server's canonical hostname, don't use "http://localhost:8080/jenkins" as the reverse proxy destination. Instead, use the canonical hostname of your server. In my case, I have a virtual host with the name "jenkins.example.org", for a server whose hostname is "devops.example.org". The ProxyPassReverse rewrite rules were not seeing "localhost" coming back, and so were just passing through the Location header "http://devops.example.org:8080/jenkins", rather than re-writing it to "http://jenkins.example.org/jenkins". 

    Now on to guarding the connection with HTTPS! 

  9. Has anyone come up with a configuration that works with HAProxy?

    1. Nevermind, operator error. The X-Forward-Proto header was not being set.

  10. 1) When you build the machine and installed Jenkins your hostname was:  "jenkins-test.example.com"

    2) You have now moved your machine to production and given it a new hostname,  "jenkins-prod1.example.com"

    3) Cause: You forgot to update the hostname in the Jenkins configuration file

    4) Solution: Visit: Manage Jenkins -> Configure System -> Jenkins Location, and update the URL to use the new hostname.

  11. Hello,

    I believe this Proxy issues has been present in Jenkins for 5+ years only a zillion of chits-chats of nonsense. Nicely done community!

    Honestly I believe sending an human to Mars it's easier than configuring Jenkins with Proxy.

    I have been trying for days on end to configure this bloody Proxy connection. All the documentation, both from Jenkins and created by users contradict each other, some "official" do not even apply because either obsolete, unclear or completely rubbish.

    I have tried to create my  "Proxy using Nginx" without success, hence I am confident that sending a man to Mars it is much easier.

    Do you know the theory of monkeys? Given them enough time, by randomly typing keys on a keyboard eventually the will rewrite Shakespeare . . . .  well  I am a monkey with lots of time to spare and I have typed lots configuration one should have worked . . . . . . . . 

    Is there is a Guru on this planet of do I have to travel to a "Galaxy far-far away" to have this problem fixed?

    This is also why I will never endorse openly OpenSource: a couple of billion configurations with few trillions suggestions none applies to your problem

  12. Does anyone have a working web.config for an IIS 8.5 reverse proxy that works with newer versions of Jenkins without giving a "It appears that your reverse proxy set up is broken" error?

    If so, could we please get it documented here?

    1. Should there be a statement here that this doesn't work with IIS 8.5 and newer versions of Jenkins?

      If anyone has it working without the error message, they sure are keeping it a secret.

       

      1. I have documented what works for me. Running Jenkins behind IIS

  13. It works for me in nginx

    server {
      listen 80;
      server_name jenkins.mydomain.com;
      location / {
        proxy_pass http://jenkins.localnet:8080;
        proxy_read_timeout  90;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
      }
    }

    The order of first proxy_pass option is important.

  14. Jenkins does not support X-Forwarded-Prefix ?
    I don`t like bind --prefix in jenkins directly, i want to use my reverse proxy ability for set X-Forwarded-* automatically.
    It can simplify jenkins configuration in my company

  15. If you encounter this message even though you ARE NOT using any reverse proxy, but Jenkins with e.g. Apache Tomcat stand alone, this can be fixed by setting the following:
    CATALINA_OPTS:
    -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true

    catalina.properties:
    org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true

    https://stackoverflow.com/questions/9719224/coding-forward-and-backward-slashes-in-tomcat-7

    Jenkins is testing proxy compatibility using background requests like the following and the encoded slashes are forbidden by default in Tomcat for security reasons. Allow them if you trust your environment and the error messages disappear.
    http://example.org.de:8080/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/http%3A%2F%2Fexample.,org%3A8080%2Fjenkins%2Fmanage/

  16. Has anyone here been able to successfully use traefik as the reverse proxy for jenkins?  I always get redirected to port 8080.  This happens even after I set jenkinsUrl to not include port 8080.

  17. We were seeing the warning about reverse proxy crop up if the session timed out while sitting on the "Manage Jenkins" page, it partially refreshes and triggers the reverse proxy check but fails it since the request wasn't authenticated.

  18. I use IIS and Jakarta(isapi_redirect.dll) forward to tomcat,

    Now I can use domain name (http://jenkins.server.me https://jenkins.server.me) normal access. but, there are tips for this problem

     

    IIS

    web url: jenkins.server.me  But, I not bound ip address.

    Host file:

    ...

    127.0.0.1    jenkins.server.me

    ...

    Jakarta(isapi_redirect.dll)

    conf

            + uriworkermap.properties

                    ...

                    /jenkins.server.me/*=jenkins

                    ...

            + workers.properties

                    ps=\

                    worker.list=..., jenkins, ...

                    ...

                    worker.jenkins.port=8009

                    worker.jenkins.host=jenkins.server.me

                    worker.jenkins.type=ajp13

                    worker.jenkins.lbfactor=1

                    ...

    Tomcat

    conf

            + server.xml

                    ...

                    <Host name="jenkins.server.me"  appBase="jenkins.server.me" unpackWARs="true" autoDeploy="true">

                            ...

                    </Host>

                    ... 

    jenkins.server.me ← this is app file.

            + ROOT ← Jenkins on this folder

     

  19. Running Jenkins on CentOS 7.4

    I recently was tasked with installing and configuring Jenkins with Apache 2.4x (httpd) on CentOS. It took a couple days to get the proxy working, and with SSL certificates.

     

    Jenkins VirtualHost Conf File
    <VirtualHost *:80>
            ServerName jenkins-ci.mydomain.org
            ServerAdmin admin@mydomain.vm
    
            Redirect / https://jenkins-ci.mydomain.org/
    </VirtualHost>
    
    <Virtualhost *:443>
            ServerName jenkins-ci.mydomain.org
            ServerAdmin admin@mydomain.vm
    
            LogLevel info
            LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
            ErrorLog /var/log/jenkins-httpd-error.log
            CustomLog /var/log/jenkins-httpd-access.log combined
    
            SSLEngine on
            SSLProxyEngine on
    
            # SSL certificate and keys. Edit paths to whereever your SSL files are located
            SSLCertificateKeyFile /etc/ssl/private/mydomain_org.key
            SSLCertificateFile /etc/ssl/certs/__mydomain_org.crt
            SSLCertificateChainFile /etc/ssl/certs/__mydomain_org.ca-bundle
    
            ProxyRequests Off
            ProxyPreserveHost On
            RewriteEngine On
            RequestHeader set X-Forwarded-Proto "https"
            AllowEncodedSlashes NoDecode
    
            ProxyPass / http://localhost:8080/ nocanon
            ProxyPassReverse / http://localhost:8080/
    
            <Proxy http://localhost:8080/*>
                    Order deny,allow
                    Allow from all
            </Proxy>
    
            ProxyTimeout 120
    </Virtualhost>

    I also run the hudson diagnostics for test my reverse proxy setup.

    Husdon Reverse Proxy Test
    curl -iL -e http://jenkins-ci.mydomain.org/manage/ http://jenkins-ci.mydomain.org/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test