Skip to end of metadata
Go to start of metadata

Plugin Information

View Azure VM Agents on the plugin site for more information.

A Jenkins Plugin to create Jenkins agents in Azure Virtual Machines (via Azure ARM template).

Supported features:

  1. Windows Agents on Azure Cloud using SSH and JNLP
    • For Windows images to launch via SSH, the image needs to be preconfigured with SSH.
    • For preparing custom windows image, refer to Azure documentation
  2. Linux Agents on Azure Cloud using SSH

How to Install

You can install/update this plugin in Jenkins update center (Manage Jenkins -> Manage Plugins, search Azure VM Agents Plugin).

You can also manually install the plugin if you want to try the latest feature before it's officially released. To manually install the plugin:

  1. Clone the repo and build:

    mvn package
  2. Open your Jenkins dashboard, go to Manage Jenkins -> Manage Plugins.
  3. Go to Advanced tab, under Upload Plugin section, click Choose File.
  4. Select azure-vm-agents.hpi in target folder of your repo, click Upload.
  5. Restart your Jenkins instance after install is completed.

Prerequisites

To use this plugin to create VM agents, first you need to have an Azure Service Principal in your Jenkins instance.

  1. Create an Azure Service Principal through Azure CLI or Azure portal.
  2. Open Jenkins dashboard, go to Credentials, add a new Microsoft Azure Service Principal with the credential information you just created.

Configure the Plugin

Add a New Azure VM Agents Cloud

  1. Within the Jenkins dashboard, click Manage Jenkins -> Configure System -> Scroll to the bottom of the page and find the section with the dropdown "Add a new cloud" -> click on it and select "Microsoft Azure VM Agents"
  2. Provide a name for the cloud (plugin will generate one for you if you leave it empty, but it's recommended to give it a meaningful name).
  3. Select an existing account from the Azure Credentials dropdown or add new "Microsoft Azure Service Principal" credentials in the Credentials Management page by filling out the Subscription ID, Client ID, Client Secret and the OAuth 2.0 Token Endpoint.
  4. Click on “Verify configuration” to make sure that the profile configuration is done correctly.
  5. Save and continue with the template configuration (See instructions below).

Add a New Azure VM Agent Template

  1. Click the "Add" in "Add Azure Virtual Machine Template" to add a template. A template is used to define an Azure VM Agent configuration, like its VM size, region, or retention time.

  2. Provide meaningful name and description for your new template.

  3. Provide one or more meaningful labels, e.g. "Windows" or "Linux". Label is used by job to determine which agents will be selected to run the job, so please make sure you give it a meaningful label.

  4. Select the desired region, and VM size from dropdown list.

  5. Select the Storage Account Type, either Standard_LRS or Premium_LRS. Note that some VM size only supports Standard_LRS.

  6. Specify the Azure Storage account name or select an existing Storage account name for storing VM's OS disk. If you choose to create a new one but leave the name blank the plugin will generate a name for you.

  7. Select the disk type between Managed Disk (recommended) or Unmanaged Disk.

  8. Select the retention strategy

    • Idle Retention Strategy. You can specify the retention time in minutes. This defines the number of minutes Jenkins can wait before automatically deleting an idle agent. Specify 0 if you do not want idle agents to be deleted automatically.
    • Pool Retention Strategy. This retention strategy help you to maintain amount of agents in a specific number. You can specify the retention time in hour and the pool size.
    • Once Retention Strategy. This retention strategy make sure to use one agent only once.

    Retention time define the time of hour before automatically deleting since the agent created. And the pool size define the agent pool size you want to maintain. If you change your cloud name, template name or most of parameters (e.g. Region, Image), we will delete the existing agents at once and provision the new one according to your new template. But if you only change your Retention Time or Pool Size, we will only scale in, scale out or do nothing for you.

  9. Select a usage option:

    • If "Utilize this node as much as possible" is selected, then Jenkins may run any job on the agent as long as it is available.
    • If "Only build jobs with label expressions matching this node" is selected, Jenkins will only build a project on this node when that project is restricted to certain nodes using a label expression, and that expression matches this node's name and/or labels. This allows an agent to be reserved for certain kinds of jobs.
  10. Select a built-in image, you can choose between Windows Server 2016 and Ubuntu 16.04 LTS. You can also choose to install some tools on the agent, including Git, Maven and Docker (JDK is always installed).

  11. Specify Admin Credentials (a username/password credentials), this is the username and password if you want to log into the agent VM.

  12. Click Verify Template to make sure all your configurations are correct, then Save.

Run Jenkins Jobs on Azure VM Agents

After you configured an Azure VM agent template, when you run a new Jenkins job, Jenkins will automatically provision a new Azure VM only if there is no executor available.

A more common scenario is you want to restrict some jobs to always be running on a particular VM agent instead of Jenkins master. To achieve that:

  1. Open your Jenkins project, under General, check "Restrict where this project can be run".
  2. In Label Expression, fill in the label you assigned to your VM template.
  3. Save and run the job, you'll see your job is running on the VM agent even if Jenkins master is free.

For how to select agent in pipeline, refer to this doc.

Use a Custom VM Image

The built-in image only has a clean Windows or Ubuntu OS and some tools like Git and Maven installed, in some cases, you may want to have more customization on the image. To use a custom image:

  1. In Image Configuration, select "Use Advanced Image Configurations".

  2. Choose between two possible alternatives:

    • Use a custom user image (provide image URL and OS type - note, your custom image has to be available into the same storage account in which you are going to create agent nodes);
    • Using any marketplace image by specifying an image reference (provide image reference by publisher, offer, sku and version). You can get the publisher, offer and sku by looking at the ARM template of that image.
  3. For the launch method, select SSH or JNLP.

    • Linux agents can be launched only using SSH.
    • Windows agents can be launched using SSH or JNLP. For Windows agents, if the launch method is SSH then check Pre-Install SSH in Windows Slave or image needs to be custom-prepared with an SSH server pre-installed.

    We recommend to use SSH rather than JNLP, for you need less init codes and get much clearer logs.

    When using the JNLP launch option, ensure the following:

    • Jenkins URL (Manage Jenkins -> Configure System -> Jenkins Location)
    • The URL needs to be reachable by the Azure agent, so make sure to configure any relevant firewall rules accordingly.
    • TCP port for JNLP agent agents (Manage Jenkins -> Configure Global Security -> Enable security -> TCP port for JNLP agents).
    • The TCP port needs to be reachable from the Azure agent launched using JNLP. It is recommended to use a fixed port so that any necessary firewall exceptions can be made.

    If the Jenkins master is running on Azure, then open an endpoint for "TCP port for JNLP agent agents" and, in case of Windows, add the necessary firewall rules inside virtual machine (Run -> firewall.cpl).

  4. For the Initialization Script, you can provide a script that will be executed after the VM is provisioned. This allows to install any app/tool you need on the agent. Please be noted you need to at least install JRE if the image does not have Java pre-installed. We prepared a sample script for Linux via SSH, Windows via SSH and Windows via JNLP. Please find details in help button.

    If you hit the storage scalability limits for your custom images on the storage account where the VHD resides, you should consider using the agent's temporary storage or copy your custom image in multiple storage accounts and use multiple VM templates with the same label within the same agent cloud.

    For more details about how to prepare custom images, refer to the below links:

    Init script should finish in 20 minutes (this time can be configured in Deployment Timeout setting of Azure Profile Configuration). It's not recommended to run complex init script, if the init script is expected to take a long time to complete, it is recommended to use a custom-prepared image.

Advanced Configurations

If you choose Use Advanced Image Configurations, you can click on Advanced button where you can find more VM configurations:

  1. Virtual Network Name, Virtual Network Resource Group Name and Subnet name: by default the VM does not belong to any virtual network, you can provide one if you want the VM to be in a virtual network for network security. Please be noted the virtual network must exist.

  2. Make VM agent IP private: by default the plugin will create a public IP for the VM so it's public accessible on internet. Check this option if you don't want the public IP to be created.

    Make VM agent IP private can make the VM more secure, but if you configured to use SSH to launch agent, Jenkins master needs to be able to access the VM. So in this case you need to also specify virtual network and subnet name so the agent and Jenkins master are in the same subnet.

  3. Network Security Group Name: add the VM to a network security group.

  4. JVM Options: specify JVM options.

  5. Number of Executors: specify the number concurrent builds that a VM agent can run at the same time.

  6. Disable template: disable this template temporarily.

Configure VM Template using Groovy Script

In some cases you may want to configure the VM template using script so it can be automated instead of manually configure it in UI. Jenkins supports groovy script that can automates such operation. Here is a sample groovy script that creates a new Azure cloud and VM template. You can run it in Manage Jenkins -> Script Console.

Configure cloud with built-in image
import com.microsoft.azure.vmagent.builders.*;

def myCloud = new AzureVMCloudBuilder()
    .withCloudName("myAzure")
    .withAzureCredentialsId("<your azure credential ID>")
    .withNewResourceGroupName("<your Resource Group Name>")
    .addNewTemplate()
        .withName("ubuntu")
        .withLabels("ubuntu")
        .withLocation("East US")
        .withVirtualMachineSize("Standard_DS2_v2")
        .withNewStorageAccount("<your Storage Account Name>")
        .addNewBuiltInImage()
            .withBuiltInImageName("Ubuntu 16.14 LTS")
            .withInstallGit(true)
            .withInstallMaven(true)
            .withInstallDocker(true)
        .endBuiltInImage()
        .withAdminCredential("<your admin credential ID>")
    .endTemplate()
    .build();

Jenkins.getInstance().clouds.add(myCloud);
Configure cloud with mutli-template of advanced images
import com.microsoft.azure.vmagent.builders.*;

def firstTemplate = new AzureVMTemplateBuilder()
    .withName("first-template")
    .withLabels("ubuntu")
    .withLocation("East US")
    .withVirtualMachineSize("Standard_DS2_v2")
    .withNewStorageAccount("<your Storage Account Name>")
    .addNewAdvancedImage()
        .withReferenceImage("Canonical", "UbuntuServer", "16.04-LTS", "latest")
        .withInitScript("sudo add-apt-repository ppa:openjdk-r/ppa -y \n" +
                        "sudo apt-get -y update \n" +
                        "sudo apt-get install openjdk-8-jre openjdk-8-jre-headless openjdk-8-jdk -y")
    .endAdvancedImage()
    .withAdminCredential("<your admin credential ID>")
    .build();

def myCloud = new AzureVMCloudBuilder()
    .withCloudName("myAzure")
    .withAzureCredentialsId("<your azure credential ID>")
    .withNewResourceGroupName("<your Resource Group Name>")
    .addToTemplates(firstTemplate)
    .addNewTemplate()
        .withName("second-template")
        .withLabels("windows")
        .withLocation("Southeast Asia")
        .withVirtualMachineSize("Standard_DS2_v2")
        .withNewStorageAccount("<your Storage Account Name>")
        .addNewAdvancedImage()
            .withReferenceImage("MicrosoftWindowsServer", "WindowsServer", "2016-Datacenter", "latest")
        .endAdvancedImage()
        .withAdminCredential("<your admin credential ID>")
    .endTemplate()
    .build();

Jenkins.getInstance().clouds.add(myCloud);
inherit existing template
import com.microsoft.azure.vmagent.builders.*;
import com.microsoft.azure.vmagent.*;

AzureVMAgentTemplate baseTemplate = new AzureVMTemplateBuilder()
    .withLocation("Southeast Asia")
    .withVirtualMachineSize("Standard_DS2_v2")
    .withStorageAccountType("Premium_LRS")
    .withNewStorageAccount("<your Storage Account Name>")
    .addNewAdvancedImage()
         .withReferenceImage("Canonical", "UbuntuServer", "16.04-LTS", "latest")
    .endAdvancedImage()
    .withAdminCredential("<your admin credential ID>")
    .build();

def myCloud = new AzureVMCloudBuilder()
    .withCloudName("myAzure")
    .withAzureCredentialsId("<your azure credential ID>")
    .withNewResourceGroupName("<your Resource Group Name>")
    .addNewTemplateLike(baseTemplate)
        .withName("inherit")
        .withLabels("inherit")
        .addNewAdvancedImageLike(baseTemplate.getAdvancedImageInside())
            .withInitScript("sudo add-apt-repository ppa:openjdk-r/ppa -y \n" +
                            "sudo apt-get -y update \n" +
                            "sudo apt-get install openjdk-8-jre openjdk-8-jre-headless openjdk-8-jdk -y")
        .endAdvancedImage()
    .endTemplate()
    .build();

Jenkins.getInstance().clouds.add(myCloud);

This sample only contains a few arguments of builder, please find all the arguments in folder builders.

Changelog

Version 0.7.4, 2018-10-18

Version 0.7.3, 2018-08-06

  • Fixed Jackson version conflicts caused by Azure client runtime lib updates JENKINS-52838

Version 0.7.2.1, 2018-08-01

  • Fixed an issue that built-in images may fail to provision

Version 0.7.2, 2018-07-25

  • Fix missing plan information for some images in reference image configuration JENKINS-52407

Version 0.7.1, 2018-05-23

  • Update Provision retry strategy in order to prevent hitting Azure request rate limit
  • Update validate logic to avoid bad template by accidentally issue
  • Change the way of generating name of VMs to resolve the same name conflict
  • Remove sleep while shutting down VMs in OnceRetentionStrategy

Version 0.7.0, 2018-04-09

Version 0.6.2, 2018-02-24

  • Use Tls1.2 to fix the bug in SSH initialization
  • Support Windows Server 1709 as SSH slave

Version 0.6.1, 2018-02-09

  • Fix location verification on non-global clouds

Version 0.6.0, 2018-01-02

  • Use deallocation instead of powerOff when configured shutdown only
  • Add built-in windows image with docker
  • Redesign verification task to improve performance
  • Fix launching issues when using performance limited vms

Version 0.5.0, 2017-11-29

  • Add support for the Managed Service Identity (MSI) as credential
  • Clean init scripts after the deployment
  • Fix some minor bugs

Version 0.4.8, 2017-11-07

  • Add Cloud and Template builder with fluent interface
  • Add pool retention strategy and once retention strategy
  • Fix bugs and improve performance
  • Maven version in built-in image update to 3.5.2
  • Add Third Party Notice

Version 0.4.7.1, 2017-08-10

  • Fixed an issue that built-in init script cannot run correctly under Linux.

Version 0.4.7, 2017-08-01

  • Built-in image support. Now besides manually fill in image reference and init script, you can select from two built-in images, Windows Server 2016 and Ubuntu 16.04 LTS.
  • Auto tool installation on VM agents. If you're using built-in image, you can choose to install Git, Maven or Docker.
  • SSH support for Windows agent. You can now use SSH to launch Windows agent.
  • Support managed disk for VM agent.
  • Allow user to specify the name for the cloud. This fixes the issue that one cloud will be ignored if you have two with same subscription ID and resource group name.
  • Various minor bug fixes.

Version 0.4.6, 2017-06-20

  • Add LogRecorder for Azure VM Agent plugin to make it easier for troubleshooting
  • Fix an issue that Jenkins crashes in some cases when CloudStatistics is enabled
  • Improve VM template creation, you're now able to select from existing resource groups and storage accounts.

Version 0.4.5.1 Beta, 2017-06-09

  • Fixed a backward compatibility issue that storage type becomes empty when reading a configuration created from an older version. (JENKINS-44750)

Version 0.4.5 Beta, 2017-06-02

  • Added the option to specify different resource group for virtual network. (JENKINS-43909)
  • Support multiple cloud profiles with the same subscription ID (JENKINS-43704)
  • Support premium storage account for VM template (JENKINS-43097)
  • Support Cloud Statistics Plugin (JENKINS-42799)

Version 0.4.4 Beta, 2017-04-12

  • Added the option to deploy VM Agents without a public IP. (JENKINS-40620)
  • Added the ability to attach a public IP for an already deployed agent. The user need to go to the Nodes management page and configure the desired Azure VM Agent.
  • Added the option to attach an existing Azure Network Security Group to the provisioned agents
  • Fixed the 'Usage' parameter. (JENKINS-42037)
  • Fixed a Null Pointer Exception while trying to fill the VM Size dropdown. (JENKINS-42853)
  • The agent password verification now allows more special characters. (JENKINS-43243)

Version 0.4.3 Beta, 2017-03-03

  • The plugin now depends on the Azure Credentials plugin
  • Existing credentials are now working when upgrading the plugin (JENKINS-42479)
  • Fixed an issue where deployments in some existing Azure Resource Groups might have failed
  • The plugin now removes the leftover empty containers after a custom-image agent was deleted.

Version 0.4.2 Beta, 2017-01-16

  • Extend support to all Azure regions and available VM sizes (JENKINS-40488)
  • Fixed an edge case where provisioned VMs were not removed after they were used (JENKINS-41330)
  • The fix will ensure that newly created resources are properly disposed, but it won't disposed of any resources deployed with the plugin before and including version 0.4.1
  • Updated the 'Max Jenkins Agents Limit' label in the cloud configuration page to reflect the quota on the number of agents the plugin is allowed to deploy in a resource group (JENKINS-41568)
  • Other minor fixes and improvements

Version 0.4.1 Beta, 2016-12-13

  • Verify that the storage account name is valid during template verification (JENKINS-40289)
  • The auto-generated storage account name is unique across Azure (JENKINS-40288)
  • The SSH session is re-established after the init script runs (JENKINS-40291)
  • Other minor fixes and improvements

Version 0.4.0 Beta, 2016-12-06

  • Initial release

41 Comments

  1. Not clear . In case if we want to use JNLP. how exactly we run command for connecting slave to jenkins. what should be done? 

    1. Windows or Linux? If Windows, you can take here as an example, and we don't support JNLP in Linux now.

  2. Is it possible to make Clean VM job run more often? 

    1. Do you means periodic clean task? It's already execute every 5 minute in 0.4.8. How often do you want?

      1. I decide to use Azure Once Retention (max cost optymization) in max moment I want to have couple tousands of VM's.

        After my jenkins job finished, it's taking even 20 min for start deleting vm

        1. Have you tested whether the clean job will take such a long time to delete vm in a smaller scale.

          I did some test but all the VMs started to deleted within 10 minutes. Can you give more detail info, or take a look in logs whether a vm that tagged to be deleted start to delete in the following periodic clean task.

           

          1. It's look like mine configuration error, when I used option "Use this node as much as it possible" that's why it's waited so long time, after change to run only with specific label, deletion time reduced.

  3. I try to create groovy script for adding new templates to existing Clouds

    I'm using builders, but it's not working for already existing Cloud, there's a possibility to pass AzureVMCloud object, code:

    AzureVMCloud cloud = Jenkins.getInstance().getCloud(cloudName);
     
    AzureVMAgentTemplate baseTemplate = new AzureVMTemplateBuilder() ...
     
     
    def myCloud = new AzureVMCloudBuilder(cloud)
    .addToTemplates(baseTemplate)
    .build();
    
    
    Jenkins.getInstance().clouds.add(myCloud);
     

     

    Error

    java.lang.UnsupportedOperationException
    	at java.util.Collections$UnmodifiableCollection.add(Unknown Source)
    	at com.microsoft.azure.vmagent.builders.AzureVMCloudBuilder.addToTemplates(AzureVMCloudBuilder.java:91)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     

    Is it possible to adding new templates to already existing Clouds? Or I need to delete and recreate it?

    1. Sorry, that's a bug. The existing Cloud return a unmodifiable template list. I will fix it ASAP

      By the way, clouds.add() will not replace the origin cloud, you need to use clouds.remove() explicitly.

  4. Hi there, guys

    I have an issue with this plugin if anyone can help me.  

    Every time I'm creating Slaves, they are active 2 hours, then the Jenkins Agent dies. After 10 min, the plugin kills my VMs. It does not matter if I'm starting jobs on these VM Slaves or not, after 2 hours they are gone.

    I've created another Slave manually and did all the necessary settings and in this case the Jenkins Agent is not dying and VM is not deleted. 

    My settings considering the retention time and the disposal of the slave are as below

    These are the logs that I'm getting. Can anyone help me with this issue, please?

     

    Started Azure VM Agents Clean Task
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: start
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: Running clean with 15 minute timeout
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: check for rqa-swar159f90
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: rqa-swar159f90 exists
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:25:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: end


    AzureVMCloudVerificationTask: getVirtualMachineCount: start
    Nov 16, 2017 6:27:25 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask getVirtualMachineCount
    AzureVMCloudVerificationTask: getVirtualMachineCount: end, currently 2 vms
    Nov 16, 2017 6:27:25 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying 2 template(s)
    Nov 16, 2017 6:27:25 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying rqa-swarm-linux in Test-AutomationApp
    Nov 16, 2017 6:27:25 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: rqa-swarm-linux verified succesfully
    Nov 16, 2017 6:27:25 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying rqa-swarm-win in Test-AutomationApp
    Nov 16, 2017 6:27:29 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: rqa-swarm-win verified succesfully
    Nov 16, 2017 6:27:29 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask execute
    AzureVMCloudVerificationTask: execute: end
    Nov 16, 2017 6:27:29 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished AzureVMCloudVerificationTask. 4,789 ms
    Nov 16, 2017 6:27:53 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:27:53 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:27:53 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:27:53 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:28:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:28:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:28:35 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started AzureVMCloudVerificationTask
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask execute
    AzureVMCloudVerificationTask: execute: start
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask execute
    AzureVMCloudVerificationTask: execute: Running verification with 15 minute timeout
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying cloud Test-AutomationApp
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: subscription Test-AutomationApp already verified
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask getVirtualMachineCount
    AzureVMCloudVerificationTask: getVirtualMachineCount: start
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask getVirtualMachineCount
    AzureVMCloudVerificationTask: getVirtualMachineCount: end, currently 2 vms
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying 2 template(s)
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying rqa-swarm-linux in Test-AutomationApp
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: template rqa-swarm-linux in Test-AutomationApp already verified
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: verifying rqa-swarm-win in Test-AutomationApp
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask verify
    AzureVMCloudVerificationTask: verify: template rqa-swarm-win in Test-AutomationApp already verified
    Nov 16, 2017 6:28:42 PM INFO com.microsoft.azure.vmagent.AzureVMCloudVerificationTask execute
    AzureVMCloudVerificationTask: execute: end
    Nov 16, 2017 6:28:42 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished AzureVMCloudVerificationTask. 6,724 ms
    Nov 16, 2017 6:29:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:29:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:29:23 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:29:23 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:29:23 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:29:23 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:29:54 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:30:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:30:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:30:52 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Agents Clean Task
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: start
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: check for rqa-swar159f90
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: Running clean with 15 minute timeout
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: rqa-swar159f90 exists
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:30:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: end
    Nov 16, 2017 6:30:52 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Agents Clean Task. 304 ms
    Nov 16, 2017 6:31:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:31:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:31:22 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:31:22 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:31:22 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:31:22 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:31:52 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:31:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call
    AzureVMAgentCleanUpTask: cleanVMs: shutting down rqa-swar159f90
    Nov 16, 2017 6:31:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent rqa-swar159f90
    Nov 16, 2017 6:31:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for rqa-swar159f90
    Nov 16, 2017 6:32:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:32:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:32:23 PM INFO com.microsoft.azure.vmagent.retry.RetryTask call
    Handle retry due to:
    java.lang.NullPointerException
    at com.microsoft.azure.vmagent.AzureVMAgent.shutdown(AzureVMAgent.java:463)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:447)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2.call(AzureVMAgentCleanUpTask.java:434)
    at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    Nov 16, 2017 6:33:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:33:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:34:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:34:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:35:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Maintainer Pool Size
    Nov 16, 2017 6:35:04 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Finished Azure VM Maintainer Pool Size. 1 ms
    Nov 16, 2017 6:35:52 PM INFO hudson.model.AsyncPeriodicWork$1 run
    Started Azure VM Agents Clean Task
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: start
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: Running clean with 15 minute timeout
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: check for rqa-swar159f90
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: rqa-swar159f90 doesnt exist
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs
    AzureVMAgentCleanUpTask: cleanVMs: node rqa-swar159f90 doesnt exist, removing
    Nov 16, 2017 6:35:52 PM FINE com.microsoft.azure.vmagent.AzureVMCloud
    AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label rqa-swar159f90
    Nov 16, 2017 6:35:52 PM FINE com.microsoft.azure.vmagent.AzureVMCloud
    AzureVMCloud: getAzureAgentTemplate: Found agent template rqa-swarm-win
    Nov 16, 2017 6:35:52 PM FINE com.microsoft.azure.vmagent.AzureVMCloud
    AzureVMCloud: getAzureAgentTemplate: Found agent template rqa-swarm-linux
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments
    Nov 16, 2017 6:35:52 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments
    Nov 16, 2017 6:35:53 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanLeakedResources
    cleanLeakedResources: deleting rqa-swar159f90NIC from resource group MyResourceGroup
    Nov 16, 2017 6:36:03 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanLeakedResources
    cleanLeakedResources: deleting rqa-swar159f90IPName from resource group MyResourceGroup

     
    1. The reason is not very clear now. The logs is very strange here and need more tests.

      Your VM was shut down by CleanUpTask, it means something set a CleanUpAction to your vm. But I don't know what function set that after 2 hours from logs.

      Another thing is something delete your VM in resource group directly but noting related in logs. Do you have any logs related to removing VMs. 

      I think it's maybe a bug in shutdown VMs, could you test a similar scenario but do not check "Shut down only after retention time".

      And what's the meaning of “created another Slave manually”. If there is another agent not created by plugin then the bugs disappeared?

       

      1. With my manually create nodes I was just trying to see if the problem is Jenkins Agent related, in a controlled way.

        Regarding the deletion, I'll put a full log below. 

        Seams like there is an issue at AzureVMAgentCleanUpTask.java:157, this leading into the error that I'm getting "Failed to get/delete deployment: {0}". 
        I think is failing when is trying to get the Deployment by using the resource group method getByResourceGroup().

        I hope this helps.

        Thank you in advance!
        log1.txt

        1.  AzureVMAgentCleanUpTask.java:157 is not a key point, it doesn't related to cleaning vms.

          Does this log from Jenkins → log → Azure VM Agent (Auto) ? This log is not full enough because some other related events like JNLP connection and deleting nodes isn't logged in this log file. Could you please give me the full log (Jenkins → log → All Jenkins logs). 

          There are 2 key problems according to this logs.

          One is why VMs going to shut down after about 1.5 to 2 hours (rqa-swar159f91 in 5:00:52 PM and rqa-swar159f90 in 5:20:52 PM). They were created in the same time but shut down in different time, so I guess it's related to connection problem.

          Another one is why they are deleted in resource group and I can only get rqa-swar159f90 deleted in 6:35:52 PM  from logs and no info about rqa-swar159f91. 

          So, if you can give me the full Jenkins logs from 4:21:05 PM to 6:35:52 PM, that will help me a lot to detect the key point of problems.

           

          1. Guys, I'm sorry for pointing this issue, but seams that Jack Chen was right and his solution resolved the deletion part. Another colleague of mine did set up another Jenkins Server and that affected my Server. 

            Sorry again and thank you a lot.

    2. Do you by any chance have multiple jenkins servers ?

       

      I saw similar problem before and turned out the VM was deleted by another Jenkins server, the other Jenkins server was cloned from the first Jenkins server, so they have same secret.key file ( so they have same Jenkins ID ). the fix is just delete the secret.key file from one Jenkins.

      https://support.cloudbees.com/hc/en-us/articles/204713750-Copying-Jenkins-home-gives-duplicate-instance-ids

      1. You were right. Thank you very much.

  5. Is there chance to add posibility to use Azure Images created from managed disks?

    I didn't find way to use Azure Image of VM, so I create JIRA task for that ( JENKINS-48076 - Getting issue details... STATUS ) - I found workaround for it, just need to copy vhd to storage after generate SAS token

    But next question, scipts which run after create VM, running with admin priviliges? (Windows case)

  6. During VM creation I get error: as below, what's strange for me, couple minutes earlier plugin get VM from Azure and even running jobs on it

     

    Unexpected exception encountered while provisioning agent jenkins-build-vm
    java.net.UnknownHostException: login.microsoftonline.com
    	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    	at java.net.PlainSocketImpl.connect(Unknown Source)
    	at java.net.SocksSocketImpl.connect(Unknown Source)
    	at java.net.Socket.connect(Unknown Source)
    	at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
    	at sun.security.ssl.BaseSSLSocketImpl.connect(Unknown Source)
    	at sun.net.NetworkClient.doConnect(Unknown Source)
    	at sun.net.www.http.HttpClient.openServer(Unknown Source)
    	at sun.net.www.http.HttpClient.openServer(Unknown Source)
    	at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
    	at sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
    	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
    	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
    	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
    	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
    	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
    	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)
    	at com.microsoft.aad.adal4j.AdalOAuthRequest.configureHeaderAndExecuteOAuthCall(AdalOAuthRequest.java:140)
    	at com.microsoft.aad.adal4j.AdalOAuthRequest.send(AdalOAuthRequest.java:83)
    	at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:80)
    	at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:818)
    	at com.microsoft.aad.adal4j.AuthenticationContext.access$100(AuthenticationContext.java:66)
    	at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:174)
    	at com.microsoft.aad.adal4j.AuthenticationContext$1.call(AuthenticationContext.java:163)
    Caused: java.util.concurrent.ExecutionException
    	at java.util.concurrent.FutureTask.report(Unknown Source)
    	at java.util.concurrent.FutureTask.get(Unknown Source)
    	at com.microsoft.azure.credentials.ApplicationTokenCredentials.acquireAccessToken(ApplicationTokenCredentials.java:145)
    Caused: java.io.IOException: java.net.UnknownHostException: login.microsoftonline.com
    	at com.microsoft.azure.credentials.ApplicationTokenCredentials.acquireAccessToken(ApplicationTokenCredentials.java:159)
    	at com.microsoft.azure.credentials.ApplicationTokenCredentials.getToken(ApplicationTokenCredentials.java:127)
    	at com.microsoft.azure.credentials.AzureTokenCredentials.getToken(AzureTokenCredentials.java:59)
    	at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:36)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:43)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at com.microsoft.azure.vmagent.AzureVMAgentPlugin$AzureTelemetryInterceptor.intercept(AzureVMAgentPlugin.java:29)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
    	at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:190)
    	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
    	at okhttp3.RealCall.execute(RealCall.java:57)
    	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
    	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171)
    	at rx.Subscriber.setProducer(Subscriber.java:211)
    	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
    	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152)
    	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138)
    	at rx.Observable.unsafeSubscribe(Observable.java:10142)
    	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    	at rx.Observable.subscribe(Observable.java:10238)
    	at rx.Observable.subscribe(Observable.java:10205)
    	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
    Caused: java.lang.RuntimeException
    	at rx.exceptions.Exceptions.propagate(Exceptions.java:58)
    	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464)
    	at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
    	at com.microsoft.azure.management.resources.implementation.ResourceGroupsInner.list(ResourceGroupsInner.java:936)
    	at com.microsoft.azure.management.resources.implementation.ResourceGroupsImpl.list(ResourceGroupsImpl.java:41)
    	at com.microsoft.azure.management.resources.implementation.DeploymentsImpl.getByName(DeploymentsImpl.java:66)
    	at com.microsoft.azure.management.resources.implementation.DeploymentsImpl.getByName(DeploymentsImpl.java:30)
    	at com.microsoft.azure.vmagent.AzureVMCloud.createProvisionedAgent(AzureVMCloud.java:604)
    Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException
    	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54)
    	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
    	at com.microsoft.azure.vmagent.AzureVMCloud.createProvisionedAgent(AzureVMCloud.java:661)
    	at com.microsoft.azure.vmagent.AzureVMCloud$3.call(AzureVMCloud.java:848)
    	at com.microsoft.azure.vmagent.AzureVMCloud$3.call(AzureVMCloud.java:825)
    	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    	at java.util.concurrent.FutureTask.run(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    	at java.lang.Thread.run(Unknown Source)
    1. Is the issue still there. It maybe some issue caused by Azure

      1. Nov 24, 2017 9:34:05 PM SEVERE com.microsoft.azure.vmagent.AzureVMCloud$3 call
        Failure creating provisioned agent 'jenkins-3fe900'
        java.net.UnknownHostException: login.microsoftonline.com
         
        Nov 24, 2017 10:09:34 PM SEVERE com.microsoft.azure.vmagent.AzureVMCloud$3 call
        Failure creating provisioned agent 'jenkins-169400'
        java.net.UnknownHostException: login.microsoftonline.com
         
        Yup, it's still
        1. It's seem some issues with your DNS.

          Could you run nslookup on you Jenkins master node to test whether resolving login.microsoftonline.com correctly or not.

  7. Hi Guys,

     

    Please avoid to use  Premium storage for Jenkins Azure Addon (sad)

    1. Is there any problem with Premium storage?

      1. We got stabbed with huge amount of storage charge. worked with Azure support for almost a week, finally figured out it's caused by the Azure VM deployment. When a VM is deployed, the addon will save a tiny powershell initial script into the same storage account for VM disk; if the VM is Premium, then the script file is also saved into premium storage account.  https://stackoverflow.com/questions/29079268/differences-between-azure-block-blob-and-page-blob  Premium page blob will be rounded up to at minimum 32G size... To make things worse, when the VM is deleted, the script file is left in storage account, and nobody thought the tiny 2K file cost anything, and they just accumulate...  

        1. I reported bug for that  JENKINS-48218 - Getting issue details... STATUS

  8. I get some strange behaviour, case:

    Build job with label 'test' → only Azure VM's use that

    I Abort job and manually delete VM (Nodes → delete)

    Restart job, silent, no VM adding process etc

    Jenkins 2.94 (all plugins updates etc

     

    Jenkins log

    AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute
    AzureVMAgentCleanUpTask: execute: Running clean with 15 minute timeout
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment jenkins-test-vm1213151128261
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 12/13/17 3:31 PM
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment jenkins-test-vm1213182314967
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 12/13/17 6:40 PM
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping
    
    Dec 13, 2017 6:50:44 PM INFO com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments
    AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments

     

    Looks like deployment deleted manually still exists from plugin perspective

    1. From my experience, if you abort task and agent then rerun, it will take a long time to start to provision new agent. It's Jenkins's schedule.

      The CleanUpTask has nothing to do with provision.

  9. I have question regarding Agents Cloud configuration, case:

    • I want max 3 VM with label eq build (they are quite expensive)
    • I want upto 20 VM's with label test - Basic A1 / A2 

    So, I don't want to made global value for 23 max vm, because than I can provision more build VM than I want (cost reduction)

     

    Can I just create two separte Cloud configs (in the same azure subscription) ? Could I expect any unusual actions? Like cleanup task delete VM's from other configuration etc?

  10. Hi there,

    We're having issues provisioning Windows agents. When we start a Windows build, Azure successfully receives instruction to create a VM and its associated resources. However, after the VM is created, the Azure plugin does not successfully download the init script (in code as REMOTE_INIT_FILE_NAME_WINDOWS within AzureVMAgentSSHLauncher) to the machine (from the configured Storage Account.) Since this .ps1 is not downloaded and ran, OpenSSH is not installed and thus a connection cannot be made. As a result, the Azure VM Agents plugin times out with the error: "AzureVMAgentSSHLauncher: connectToSsh: Got exception while connecting to remote host. Will be trying again after 1 minute timeout: socket is not established" and this is repeated a number of preconfigured seconds until a final exception is caught with the error: "timeout: socket is not established"


    What can we do to remedy this?

    1. Why does the plugin attempt to make a connection on line 107 of AzureVMAgentSSHLauncher.java before it has downloaded OpenSSH to the VM so that a connection can be made, on line 149 and below of AzureVMAgentSSHLauncher.java?

      1. No, actually SSH will be installed before executing AzureVMAgentSSHLauncher.java. We use Azure VM extension to install SSH and the init.ps1 is only going to install java.

        Which version of Windows  slave were you using. There is a known issue that we don't support windows server 1709 now.

        1. Hi Chenyang,

          Stepping through AzureVMAgentSSHLauncher, I noticed that it is attempting to make a call to the connectToSsh function on line 107. This is where our problem occurs, which is before SSH is installed further down on line 149 and below.

          We are using an Image Reference: Publisher: MicrosoftVIsualStudio, Offer: VisualStudio, Sku: VS-2017-Ent-Latest-WS2016, Version: Latest

          1. You shouldn't install SSH inside initScript by yourself. If you want to use SSH, you should check "Pre-Install SSH in Windows Slave" in the configuration and thus we will install SSH for you (Using VM Extension rather than initScript) before launching slave via SSH.

            1. Hi Chenyang,

              We are not installing ssh inside the init script. As you stated, we checked "Pre-Install SSH in Windows Slave." When this occurs, I watch and see that the plugin successfully uploads the .ps1 initScript to the Azure Storage Account resource. However, it is never executed.

              1. Maybe the issue related to  JENKINS-49725 - Getting issue details... STATUS

  11. Hello,

    I have one question and one issues.

    My issue is sometimes when I load up the Jenkins Configuration page the Azure Cloud does not pull in all the fields correctly. They are blank and do not reload. This could be an issue with the cache. I have clicked save which then saves the Azure Cloud with those blank fields (region, credential) and breaks the setup.

     

    My question is for the Agent Pool retention I was wondering if a feature request could be logged to add advanced configuration to that so it can be scaled up and down by time. My example is at night on probably only need 1 agent for say windows but during the day I need a few more. It would be useful to have an automatic scale up and down once the time is hit and the agent is idle (for scale down). If this can be accomplished another way say a jenkins job that uses the API I could look into that but an example would be helpful.

  12. Hello,

    I'm getting some strange behaviour with vm's created with plugin when vm should be stopped not deleted.

    I create 2 vm's (both have diff labels), when I today run job from one of them: 

    jenkins-81a2c0 is offline

    and plugin creating next vm for me, but from my view should turn on existing one. Also, when I go to jenkins → nodes, both machines are only offline, but should be offline and suspended.

    Only way is to delete existing offline vm's and recreate them.

     

    I check logs and looks like there was error during shutdown:

     

    Mar 09, 2018 8:08:53 PM com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy check
    INFO: AzureVMCloudRetensionStrategy: check: Idle timeout reached for agent: jenkins-81a2c0, action: shutdown
    Mar 09, 2018 8:08:53 PM com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1 call
    INFO: AzureVMCloudRetensionStrategy: going to idleTimeout agent: jenkins-81a2c0
    Mar 09, 2018 8:08:53 PM com.microsoft.azure.vmagent.AzureVMAgent shutdown
    INFO: AzureVMAgent: shutdown: shutting down agent jenkins-81a2c0
    Mar 09, 2018 8:08:53 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    INFO: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for jenkins-81a2c0
    Mar 09, 2018 8:08:53 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
    WARNING: Computer.threadPoolForRemoting [#724] for jenkins-81a2c0 terminated
    java.nio.channels.ClosedChannelException
    	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
    	at hudson.remoting.Channel.close(Channel.java:1446)
    	at hudson.remoting.Channel.close(Channel.java:1399)
    	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746)
    	at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:99)
    	at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:664)
    	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    	at java.util.concurrent.FutureTask.run(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    	at java.lang.Thread.run(Unknown Source)
    
    Mar 09, 2018 8:08:53 PM com.microsoft.rest.interceptors.LoggingInterceptor log
    INFO: <-- HTTP FAILED: java.net.SocketException: Connection reset by peer: socket write error
    Mar 09, 2018 8:08:53 PM okhttp3.internal.platform.Platform log
    INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path?
    Mar 09, 2018 8:08:55 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
    INFO: Accepted JNLP4-connect connection #24 from /10.216.0.5:52809
    Mar 09, 2018 8:08:58 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
    WARNING: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/172.29.0.107:49970 remote=10.216.0.5/10.216.0.5:52809]] / Computer.threadPoolForRemoting [#728] for jenkins-81a2c0 terminated
    java.nio.channels.ClosedChannelException

     

    After that VM is useless (sad)

     

    1. I found jirra issue  JENKINS-49021 - Getting issue details... STATUS  at this point this behavior it's blocker for us. 

  13. Hi,

    Would you be able to add support for Dsv3-series family VM's? 

    Thanks

     

    1. It's already be supported. We list vm size through azure sdk.

  14. Latest plugin version:

     

    AzureVMCloud: provision: agent computer eligible for reuse jenkins-e8ed10
    Apr 25, 2018 10:55:58 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-e8ed10
    Apr 25, 2018 10:55:58 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
    AzureVMManagementServiceDelegate: virtualMachineExists: jenkins-e8ed10 exists
    Found existing node, starting VM jenkins-e8ed10
    Apr 25, 2018 10:55:58 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision
    AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s)
    Apr 25, 2018 10:55:58 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate startVirtualMachine
    AzureVMManagementServiceDelegate: startVirtualMachine: jenkins-e8ed10
    
    Apr 25, 2018 10:58:28 AM INFO com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy check
    AzureVMCloudRetensionStrategy: check: Idle timeout reached for agent: jenkins-e8ed10, action: shutdown
    Apr 25, 2018 10:58:28 AM INFO com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1 call
    AzureVMCloudRetensionStrategy: going to idleTimeout agent: jenkins-e8ed10
    
    Azure Cloud: waitUntilOnline: for agent jenkins-e8ed10
    Apr 25, 2018 10:58:53 AM INFO com.microsoft.azure.vmagent.AzureVMCloud waitUntilJNLPNodeIsOnline
    Azure Cloud: waitUntilOnline: node is alive , result success
    Apr 25, 2018 10:58:53 AM INFO com.microsoft.azure.vmagent.AzureVMCloud$2 call
    Remove suspended status for node: jenkins-e8ed10
    Apr 25, 2018 10:58:53 AM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: Add suspended status for node jenkins-e8ed10
    Apr 25, 2018 10:58:53 AM INFO com.microsoft.azure.vmagent.AzureVMAgent shutdown
    AzureVMAgent: shutdown: shutting down agent jenkins-e8ed10
    Apr 25, 2018 10:58:53 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine
    AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for jenkins-e8ed10

     

    and keep looping, vm limit is reached, so it's gonna be inifinit. Atm our CI is blocked by this (sad)