Skip to end of metadata
Go to start of metadata

Plugin Information

View Node sharing executor on the plugin site for more information.

Share machines as Jenkins agents across multiple Jenkins masters.

Requirements

  • The nodes are connected to the individual Jenkins masters so builds can be executed there as if those nodes would be good old Jenkins nodes.
  • The node to use is determined by evaluating Jenkins labels.
  • Nodes are use exclusively by individual Jenkins masters.
  • In case there is no matching node available at the time build is scheduled the request will be queued and dispatched in FIFO fashion.

Basics

In order to facilitate fair sharing of agents, additional Jenkins master is needed to serve the role of an Orchestrator, leasing the Nodes to Executor Jenkinses.

The Executor Jenkins is an ordinary Jenkins with Shared Nodes cloud defined globally permitting it to lease, connect and utilize nodes from a shared pool. Such Jenkins can use static slaves or other clouds in the same time.

The Orchestrator Jenkins is a dedicated instance to instrument the sharing. It is expected not to have any static nodes, clouds or jobs defined as it is using the notion of computers, executors, builds and queue items to implement the sharing. It also serve the purpose of grid visualization UI for read only clients. It is not meant to be executing any other workload.

Config repository is a git based record of the current configuration of the pool the Orchestrator (and to some extend the Executors) takes as an input. Ideally, the Orchestrator should have no other user configuration except for what is in the config repository. In the name of reliability, the config repository hold the full list of Jenkins masters authorized to reserve from a particular pool. The repository, or the permission to commit there, is the access control mechanism for the pool management.

The Node is a host used for sharing. It have several representations in the Node Sharing Grid:

  • Actual Jenkins agent connected by the cloud implementation to the Executor Jenkins
  • The placeholder computer in Orchestrator.
    • Note this is not a real Jenkins computer capable of executing anything except for dummy reservation tasks. It does not even have a remoting channel open to the actual host despite it appears online.
  • The record in config repository containing the node definition.

Setup

Config Repo

  • Create git repository cloneable from both Orchestrator and Executor Jenkinses.
  • Populate it with the grid definition.

Orchestrator

  • Create minimalistic Jenkins deployment
  • Install node-sharing-orchestrator plugin including its dependencies
  • Start Jenkins with following java properties
    • -Dcom.redhat.jenkins.nodesharingbackend.Pool.ENDPOINT=cloneable_git_url_pointing_to_config_repo
    • -Dcom.redhat.jenkins.nodesharingbackend.Pool.USERNAME=name_of_the_REST_user
    • -Dcom.redhat.jenkins.nodesharingbackend.Pool.PASSWORD=password_of_the_REST_user
  • Verify no Administrative Monitor warnings are issued once Jenkins is started
  • Configure dedicated automation account to receive incoming REST calls granting it permission named NodeSharing.Reserve. No real user should be granted this permission. How to do this is specific to particular authorization strategy used.
    • Example Jenkins Configuration as Code definition:

      unclassified:
        location:
          # Needs to be set correctly so orchestrator knows its own url
          url: https://ci.example.com/orchestrator
      
      jenkins:
        numExecutors: 0
        quietPeriod: 0
        slaveAgentPort: -1
        securityRealm:
          local:
            users:
              - id: "admin"
                password: "secret"
              - id: "nodesharing"
                password: "secret too"
        # Set necessary permission for 'nodesharing' account as well for anonymous
        # user so executor users can use orchestrator as a dashboard
        authorizationStrategy:
          globalMatrix:
            grantedPermissions:
            - "Overall/Administer:admin"
            - "Overall/Read:anonymous"
            - "Job/Read:anonymous"
            - "Overall/Read:nodesharing"
            - "Job/Read:nodesharing"
            - "NodeSharing/Reserve:nodesharing"
      

      (Note it requires configuration-as-code and configuration-as-code-support plugins installed to execute, plus all the plugins needed by the declaration itself)

Executors

  • Install node-sharing-executor plugin including its dependencies on instances to utilize the pool
  • Note that node definitions in config repo can refer to further plugins that needs to be installed on executors too.
  • Add Shared Nodes cloud specifying cloneable git url pointing to config repo and credentials for the rest user.
  • Create SSH credentials used to connect the machines. Their ids needs to match those in config repo nodes.
  • Configure dedicated automation account to receive incoming REST calls granting it permission named NodeSharing.Reserve. No real user should be granted this permission. How to do this is specific to particular authorization strategy used
  • Example Jenkins Configuration as Code definition:

    unclassified:
      location:
        # Needs to be set correctly to executor knows its own url
        url: https://ci.example.com/jenkins
    
    jenkins:
      # Dedicated user needed to receive REST calls from orchestrator
      securityRealm:
        local:
          users:
            - id: "nodesharing"
              password: "nodesharing"
      authorizationStrategy:
        globalMatrix:
          grantedPermissions:
          - "Overall/Read:nodesharing"
          - "Job/Read:nodesharing"
          - "NodeSharing/Reserve:nodesharing"
    
      # Cloud hooking executor to the pool
      clouds:
        - nodeSharing:
            configRepoUrl: https://git.example.com/team/node-sharing-config-repo.git
            orchestratorCredentialsId: "node-sharing-rest-password-id"
    
    credentials:
      system:
        # Any credentials we are referring to from node definitions
        domainCredentials:
        - credentials:
          - usernamePassword:
              scope: SYSTEM
              id: "node-sharing-rest-password-id"
              username: "nodesharing"
              password: "secret"
              description: "Rest credential for node sharing"
    

    (Note it requires configuration-as-code and configuration-as-code-support plugins installed to execute, plus all the plugins needed by the declaration itself)


11 Comments

  1. Will this plugin work if used on a Jenkins  Windows setup?  exclusively windows environment?

    1. We have tested Windows nodes but we are unsure about the rest of the infra. Even the tests are disabled on windows as that is a platform we do not target for executors or orchestrator. Despite the fact I do not see a reason why it would not work conceptually, some work will likely be required.

  2. I reviewed the Nodes configuration for Windows Slave examples from the unit test.  They are using SSH Slave plugin and SSH Launcher to launch the Slave.  My question is how will this work for Windows where SSH client does not exist.  

    The concept of Sharing Node from a single Orchestrator is excellent, but there isn't much point for NIX machines where there are many solution which leverage Container to solve the Slave scaling problem.  On the other hand, this is extremely useful for Windows Slaves.

  3. Actualy our main motivation was to be able to share a resource with a bit exotic or expensive attributes like Solaris boxes, AIX, RHEL on PowerPC etc. MS Windows is also shareable, one way how to achieve this is to run OpenSSH from CygWin environment - it run excelent.

    1. Hi Pavel, thanks for the explanation.  OpenSSH from CygWin doesn't really work for Windows.  While, for controlling the machine, this implementation will work.  However, the nature of using Windows machine typically is intended to use for handling .NET build environment and VSBuild doesn't work very well in in CygWin.  


      I think the ideal of sharing node is awesome and with this feature, Jenkins can expend is community to Windows developers.  I look forward to any future update.  Thanks.

      1. The problem with and ssh alternatives for windows is they all connect from the client side (JNLP, Windows Service). This does not permit the nodesharing to be in control of a) when the machine will (not) connect and b) which executors jenkins it will connect to.


        One way to implement this I envision is, nodesharing provide a dedicated launcher that would be referred to from node configs for windows. That launcher would be executed once the machine is reserved from an Executor side and "notify" the host where to connect to. The tricky part is crafting the windows service that would be listening to such notifications and start the agent as instructed.

      2. BTW, would RDP be a suitable means of starting the agent suitable for your needs?

  4. I don't think there is a blocker... if you have a Jenkins deployment where you are satisfied with your MS Windows nodes setup/connection, you can use the same for NodeSharing as well. Just put the definition of a shared node like you have it in the standalone environment and use the same set of plugins on your Executor part. Orchestrator only serves as a storage for nodes configuration setting (see <repo>/nodes/*.xml) in that case. Our deployment depends on SSH Slave plugin (better to say our nodes XML definition utilizes this plugin for their connection from Jenkins master), but it doesn't mean this is the only way how to connect the slave actualy.

  5. I'm not sure if this is really the place for support questions so happy to be directed elsewhere...

    1. Plugins configuration-as-code, configuration-as-code-support and matrix-auth seem to be a required dependencies.  The NodeSharing/Reserve permission is not exposed in the Jenkins UI Matrix-based Security table.  Using "Logged-in users can do anything" results in complaints about the node sharing user not being assigned the NodeSharing/Reserve permission.
    2. In the executor log I am getting the following error. A similar trace is reported in the UI when testing the configured credentials.  I am using the pam-auth plugin and the user/pass are valid on the executor and orchestrator.

      SEVERE: Timer task com.redhat.jenkins.nodesharingfrontend.WorkloadReporter@23fc3a15 failed
      java.lang.IllegalArgumentException: Name may not be null
              at org.apache.http.util.Args.notNull(Args.java:54)
              at org.apache.http.message.BasicHeader.<init>(BasicHeader.java:61)
              at com.redhat.jenkins.nodesharing.RestEndpoint.getCrumbHeader(RestEndpoint.java:260)
              at com.redhat.jenkins.nodesharing.RestEndpoint.executeRequest(RestEndpoint.java:167)
              at com.redhat.jenkins.nodesharingfrontend.Api.reportWorkload(Api.java:134)
              at com.redhat.jenkins.nodesharingfrontend.WorkloadReporter.doRun(WorkloadReporter.java:104)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)

    1. Re #1: configuration-as-code, configuration-as-code-support should be needed only when you actually use JCasC to set things up. matrix-auth come into play because it is used in the JCasC definition. Let me adjust the docs.


  6. For point #2 I found it was an issue in the configuration repository.  I have opened a PR with a change: https://github.com/jenkinsci/node-sharing-plugin/pull/107