Build your own Hosted VSTS Agent Cloud: Part 1 – Build

I love Visual Studio Team Services. VSTS allows me to focus on building my CI/CD pipeline, running tests and managing my project without having to worry about installing updates or applying patches to the platform. VSTS is always on, always up to date. An important part of VSTS is the Hosted Agent Pool. Agents are used to run builds, tests and releases. They do the actual work. The Hosted Pool consists of a series of Virtual Machines that are managed by Microsoft. These VMs contain all kinds of development tools ranging from Visual Studio to Open Source tools and Docker. Microsoft tests and upgrades these VMs, so you don’t have to worry about a thing.

And that’s great. Except for when it isn’t. The set of software on the agents is predefined by Microsoft. If you need something else to run your build or release you’re on your own. Each time you run a build or release, you get a brand new VM. This means you must wait for the new VM to boot and become available. It also means there is absolutely no caching. Cloning your Git repo must be done on each build. Running npm install always means that all packages must be downloaded. The same for Docker, all base images must be downloaded each time you need them.

Running your own Agent can help you solve these issues. You can install whatever you need, have immediate access and reuse your agent between builds and releases. You can also choose the VM size you are willing to pay for to speed up your builds. But you must maintain it yourself which takes time and suddenly makes you worry about Virtual Machines, OS patches and continually upgrading your tools. For me, that’s not what I want to do on a day to day basis.

In the following blog posts I want to take a semi-managed approach. What if you could reuse all the work that Microsoft puts into the Hosted Agent Pool and use that to run your own VSTS Agent Pool. In this post I’ll introduce Packer and show you how to build your own image. In the next post we’ll look at deploying these custom images and finally we’ll setup a CI/CD pipeline on VSTS that does all this fully automated.

TL;DR; You can find all the scripts you need on GitHub to setup your own pipeline: https://github.com/WouterDeKort/VSTSHostedAgentPool

Introducing Packer

Microsoft uses Packer to build VM images that they then reuse to create Virtual Machine as part of the Hosted Agent Pool. Packer can build images running Windows and Linux on platforms like Azure, AWS and VMWare. You can view the Packer config that Microsoft uses at https://github.com/Microsoft/vsts-image-generation where the whole Agent config is open source. To get started using Packer, I would recommend setting up your pipeline with a simpler image and then switching to the full image. This saves time and makes it easier to iterate on your configuration without having to wait for the enormous Microsoft image to build.

Packer uses a JSON based configuration file that you pass to the Packer executable and that results in a complete image. The following shows a very simple Packer file that uses Azure to build a Windows Server 2016 image that’s sysprepped and ready to go:

{

   "variables": {

      "client_id": "{{env `ARM_CLIENT_ID`}}",
  
      "client_secret": "{{env `ARM_CLIENT_SECRET`}}",

      "subscription_id": "{{env `ARM_SUBSCRIPTION_ID`}}",

      "tenant_id": "{{env `ARM_TENANT_ID`}}",

      "object_id": "{{env `ARM_OBJECT_ID`}}",

      "location": "{{env `ARM_RESOURCE_LOCATION`}}",

      "managed_image_resource_group_name": "{{env `ARM_IMAGE_RESOURCE_GROUP_NAME`}}",

      "managed_image_name": "{{env `ARM_IMAGE_NAME`}}"

   },

   "builders": [{

      "type": "azure-arm",

      "client_id": "{{user `client_id`}}",

      "client_secret": "{{user `client_secret`}}",

      "subscription_id": "{{user `subscription_id`}}",

      "object_id": "{{user `object_id`}}",

      "tenant_id": "{{user `tenant_id`}}",

      "location": "{{user `location`}}",

      "vm_size": "{{user `vm_size`}}",

      "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",

      "managed_image_name": "{{user `managed_image_name`}}",

      "os_type": "Windows",

      "image_publisher": "MicrosoftWindowsServer",

      "image_offer": "WindowsServer",

      "image_sku": "2016-Datacenter",

      "communicator": "winrm",

      "winrm_use_ssl": "true",

      "winrm_insecure": "true",

      "winrm_timeout": "4h",

      "winrm_username": "packer"

   }],

   "provisioners": [{

      "type": "powershell",

      "inline": [

         "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",

         "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit",

         "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10 } else { break } }"

      ]

   }]

}

The variables section defines all the parameters you can pass on the command line or through environment variables. Builders then define how you want to create your VM image. In this sample, you configure Packer to use Azure to build the VM image and store it as a managed image in an Azure resource group. The base image used is Windows Server 2016 and WinRM is used to configure your VM. You can have multiple Builds that Packer will run in parallel producing multiple images for different environments.

The provisioners section is where the real work happens. Here you define the steps that install software on your VM and configure it the way you want. In this sample, we run only one PowerShell script that syspreps your VM. When creating a Windows image, make sure to always add this step as the last step. If you forget to do that, you end up with a non-sysprepped image. Creating a VM from that image, results in a VM that will fail to start. This sample is based on Windows, but the same constructs apply to Linux. For Linux add steps that run sh scripts and copy files to your VM.

What Packer on Azure does is create a temporary resource group of the form packer-resource-group-*. In this resource group, Packer creates a Virtual Machine that’s going to be the base for your image. When creating a Windows image, an Azure Key Vault is created that’s used for a secure WinRM connection to your Windows VM. After creating the VM, your scripts are executed. Once this is finished, Packer closes your VM and creates an image from the VMs VHD. This image is the output and that’s what you can use to create as many VMs as you need.

Running a local Packer build

To get a feeling for how Packer works I want to take you through a simple Packer build that you can run locally.

Begin by installing Packer. Packer has distributions for Linux, Mac and Windows. You can find your installer here.

Save the previous sample JSON script to a file named windows.json (or get it here on GitHub). Then create a second file named packersettings.json with the following content:

{

   "client_id": "<value of $sp.applicationId>",

   "client_secret": "P@ssw0rd!",

   "tenant_id": "<value of $sub.TenantId>",

   "subscription_id": "value of $sub.SubscriptionId",

   "object_id": "<value of $sp.Id>",

   "location": "<Location, for example West Europe>",

   "managed_image_resource_group_name": " myResourceGroup ",

   "managed_image_name": "windows-image"

}

You need to define a couple of variables for a secure connection between Packer and Azure. How to use Packer to create Windows virtual machine images in Azure shows the step you need to take. It comes down to running the following script and storing the values in your packersettings.json file:

$rgName = "myResourceGroup"

$location = "West Europe"

New-AzureRmResourceGroup -Name $rgName -Location $location

$sp = New-AzureRmADServicePrincipal -DisplayName "Azure Packer" `

 -Password (ConvertTo-SecureString "P@ssw0rd!" -AsPlainText -Force)

Sleep 20

New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId

$sub = Get-AzureRmSubscription

$sub.TenantId

$sub.SubscriptionId

Now that you have the template and settings file you can run the following command to let Packer build your image:

packer build -var-file=packersettings.json windows.json

This is going to take some time. Packer needs to deploy a Key Vault, a new VM, run your scripts, power off the VM, capture an image from it and clean up the temporary resources. A simple image can easily take 20 minutes. The full VSTS image will take 7-8 hours.

If something goes wrong, you can debug your build by setting the PACKER_LOG environment variable to 1: $env:PACKER_LOG=1. If you want to inspect the temporary Packer VM in its intermediate state, add the following argument to your packer build command: -on-error=abort. Passing this argument stops Packer from deleting the VM, allowing you to remote desktop into it and see what went wrong.

After packer build finishes, you can use the Azure portal to view the image that Packer created.

Create a VM from the Azure portal
You can view the managed image that Packer builds and create a new VM from it in the portal

Click on Create VM to enter the standard wizard to create a new Azure VM. The only difference is that instead of using one of the default VM images, you’re now using your own custom image. If those steps succeed, you have created your first custom Packer build image and a resulting VM!

Something that bit me the first time I tried this was that I forgot to add the sysprep step as the last step in my image build. If you mis that step, you will get an image and be able to create a VM, but it will never boot.

What’s next

In this post, you’ve been introduced to Packer, created your first custom image and build a new VM from that image. In the next post I’ll show you PowerShell scripts for building the image, creating the VM and installing the VSTS agent. I will then show you how to setup a CI/CD pipeline that automate these tasks for you.

Share

10 Responses

  1. Hi Wouter de Kort,
    thank you for your Blog Post. I am pretty new to Azure and I am currently working on a pretty similar Project.
    My Question is: “What is the advantage of using Packer instead of an ARM Template with a Custom Script Extension which installs the newest Enviroment for my customized VM.

    Regards from Germany
    Johannes

    • Hi Johannes,

      good question! Packer has the ability to target multiple platforms. For example, by adding a builder to your script you could create the same VM for AWS or VMWare. ARM is used only in Azure. Now, if you already know that Azure is your only target that’s not a problem of course. ARM with Custom Scripts could be a solution. However, one thing I liked about Packer is how easy it is to have steps in between that reboot the VM and than continue installing scripts. I’m not sure if ARM allows me to do that. Another option in Azure is using DevTest labs with artifacts. I don’t think one choice is absolutely better then the other. It depends on your needs and what’s available. In this scenario, I went along with the Packer choice that Microsoft already made so I could reuse their work. If I had to start from scratch building VM Agents I think I would use DevTest Labs.

      Does that answer your question a bit?

      Thanks,
      Wouter

      • We do a slightly different approach where we use an ARM template to provision a VM with Microsoft’s latest Server 2016/VS2017 VM image and configure our custom setup with Desired State Configuration defined in Azure Automation. We can setup a new build machine soup to nuts in about an hour. This has eliminated any need for us to take sysprep’d image snapshots since it’s relatively quick to just start from scratch.

  2. This seems a very promising solution to a problem I was facing (I want as little maintenance as possible but still have custom software to use during my builds/releases). So I started off with the above steps and hit a problem immediately, after 4 hours the winrm timeout is reached.

    I thought i’d use a faster VM size (D2_V2) but same issue occurred. Then I though i’d remote desktop to the machine but can’t seem to find any indication of what username/password combination to use for that. Any suggestions?

    • Hi Jeroen,

      Are you deploying the small image Packer config? I’ve found it easiest to use a very small Packer config to test if all my settings are correct. I also ran into WinRM timeouts and increasing it to 4 hours fixed it for me. Not sure what’s happening in your case. Have you made sure that all your resources are created in the same region? Especially for the bigger image configurations, I noticed that running the packer build command on a VM in Azure also speeds up the whole process.

      Regarding Remote Desktop. I also wasn’t able to figure out what the Packer username/password is. What I did was use the Azure portal to reset the configuration and supply my own username and password. After that I can successfully Remote Desktop to the machine.

      Regards,
      Wouter

      • Hi Wouter,

        I finally got around to picking this up again. The problem turned out to be firewall related. IT had just installed a new firewall in our satellite office which was blocking outgoing WinRM traffic. And it turns out that the generated packer username/password combo is displayed if you start packer with the debug command line option.

        Great blog series btw. looking forward to replace our mishmash of build and release servers with a nice standardized scalable and automatable set of uniform ones.

        Regards,
        Jeroen

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.