Build your own Hosted VSTS Agent Cloud: Part 1 - Build

2018-02-25

I love Visual Studio Team Services. VSTS allows me to focus on building my CI/CD pipeline, running tests and managing my project without having to worry about installing updates or applying patches to the platform. VSTS is always on, always up to date. An important part of VSTS is the Hosted Agent Pool. Agents are used to run builds, tests and releases. They do the actual work. The Hosted Pool consists of a series of Virtual Machines that are managed by Microsoft. These VMs contain all kinds of development tools ranging from Visual Studio to Open Source tools and Docker. Microsoft tests and upgrades these VMs, so you don’t have to worry about a thing.

And that’s great. Except for when it isn’t. The set of software on the agents is predefined by Microsoft. If you need something else to run your build or release you’re on your own. Each time you run a build or release, you get a brand new VM. This means you must wait for the new VM to boot and become available. It also means there is absolutely no caching. Cloning your Git repo must be done on each build. Running npm install always means that all packages must be downloaded. The same for Docker, all base images must be downloaded each time you need them.

Running your own Agent can help you solve these issues. You can install whatever you need, have immediate access and reuse your agent between builds and releases. You can also choose the VM size you are willing to pay for to speed up your builds. But you must maintain it yourself which takes time and suddenly makes you worry about Virtual Machines, OS patches and continually upgrading your tools. For me, that’s not what I want to do on a day to day basis.

In the following blog posts I want to take a semi-managed approach. What if you could reuse all the work that Microsoft puts into the Hosted Agent Pool and use that to run your own VSTS Agent Pool. In this post I’ll introduce Packer and show you how to build your own image. In the next post we’ll look at deploying these custom images and finally we’ll setup a CI/CD pipeline on VSTS that does all this fully automated.

TL;DR; You can find all the scripts you need on GitHub to setup your own pipeline: https://github.com/WouterDeKort/VSTSHostedAgentPool

This post is the first in a series of 4:

Introducing Packer

Microsoft uses Packer to build VM images that they then reuse to create Virtual Machine as part of the Hosted Agent Pool. Packer can build images running Windows and Linux on platforms like Azure, AWS and VMWare. You can view the Packer config that Microsoft uses at https://github.com/Microsoft/vsts-image-generation where the whole Agent config is open source. To get started using Packer, I would recommend setting up your pipeline with a simpler image and then switching to the full image. This saves time and makes it easier to iterate on your configuration without having to wait for the enormous Microsoft image to build.

Packer uses a JSON based configuration file that you pass to the Packer executable and that results in a complete image. The following shows a very simple Packer file that uses Azure to build a Windows Server 2016 image that’s sysprepped and ready to go:

{
   "variables": {
      "client_id": "{{env `ARM_CLIENT_ID`}}",
      "client_secret": "{{env `ARM_CLIENT_SECRET`}}",
      "subscription_id": "{{env `ARM_SUBSCRIPTION_ID`}}",
      "tenant_id": "{{env `ARM_TENANT_ID`}}",
      "object_id": "{{env `ARM_OBJECT_ID`}}",
      "location": "{{env `ARM_RESOURCE_LOCATION`}}",
      "managed_image_resource_group_name": "{{env `ARM_IMAGE_RESOURCE_GROUP_NAME`}}",
      "managed_image_name": "{{env `ARM_IMAGE_NAME`}}"
   },
   "builders": [{
      "type": "azure-arm",
      "client_id": "{{user `client_id`}}",
      "client_secret": "{{user `client_secret`}}",
      "subscription_id": "{{user `subscription_id`}}",
      "object_id": "{{user `object_id`}}",
      "tenant_id": "{{user `tenant_id`}}",
      "location": "{{user `location`}}",
      "vm_size": "{{user `vm_size`}}",
      "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
      "managed_image_name": "{{user `managed_image_name`}}",
      "os_type": "Windows",
      "image_publisher": "MicrosoftWindowsServer",
      "image_offer": "WindowsServer",
      "image_sku": "2016-Datacenter",
      "communicator": "winrm",
      "winrm_use_ssl": "true",
      "winrm_insecure": "true",
      "winrm_timeout": "4h",
      "winrm_username": "packer"
   }],
   "provisioners": [{
      "type": "powershell",
      "inline": [
         "if(Test-Path $Env:SystemRoot\windows\system32\Sysprep\unattend.xml)
         {
             rm $Env:SystemRoot\windows\system32\Sysprep\unattend.xml -Force
          }",
         "& $env:SystemRoot\System32\Sysprep\Sysprep.exe /oobe /generalize /quiet /quit",
         "while($true) { $imageState = Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE\_STATE\_GENERALIZE\_RESEAL\_TO\_OOBE')
         {
             Write-Output $imageState.ImageState; Start-Sleep -s 10 } else { break }
           }"
      ]
   }]
}

The variables section defines all the parameters you can pass on the command line or through environment variables. Builders then define how you want to create your VM image. In this sample, you configure Packer to use Azure to build the VM image and store it as a managed image in an Azure resource group. The base image used is Windows Server 2016 and WinRM is used to configure your VM. You can have multiple Builds that Packer will run in parallel producing multiple images for different environments.

The provisioners section is where the real work happens. Here you define the steps that install software on your VM and configure it the way you want. In this sample, we run only one PowerShell script that syspreps your VM. When creating a Windows image, make sure to always add this step as the last step. If you forget to do that, you end up with a non-sysprepped image. Creating a VM from that image, results in a VM that will fail to start. This sample is based on Windows, but the same constructs apply to Linux. For Linux add steps that run sh scripts and copy files to your VM.

What Packer on Azure does is create a temporary resource group of the form packer-resource-group-*. In this resource group, Packer creates a Virtual Machine that’s going to be the base for your image. When creating a Windows image, an Azure Key Vault is created that’s used for a secure WinRM connection to your Windows VM. After creating the VM, your scripts are executed. Once this is finished, Packer closes your VM and creates an image from the VMs VHD. This image is the output and that’s what you can use to create as many VMs as you need.

Running a local Packer build

To get a feeling for how Packer works I want to take you through a simple Packer build that you can run locally.

Begin by installing Packer. Packer has distributions for Linux, Mac and Windows. You can find your installer here.

Save the previous sample JSON script to a file named windows.json (or get it here on GitHub). Then create a second file named packersettings.json with the following content:

{
   "client_id": "<value of $sp.applicationId>",
   "client_secret": "P@ssw0rd!",
   "tenant_id": "<value of $sub.TenantId>",
   "subscription_id": "value$sub.SubscriptionId",
   "object_id": "<value of $sp.Id>",
   "location": "<Location, for example West Europe>",
   "managed_image_resource_group_name": " myResourceGroup ",
   "managed_image_name": "windows-image"
}

You need to define a couple of variables for a secure connection between Packer and Azure. How to use Packer to create Windows virtual machine images in Azure shows the step you need to take. It comes down to running the following script and storing the values in your packersettings.json file:


$rgName = "myResourceGroup"
$location = "West Europe"

New-AzureRmResourceGroup -Name $rgName -Location $location

$sp = New-AzureRmADServicePrincipal -DisplayName "Azure Packer" -Password (ConvertTo-SecureString "P@ssw0rd!" -AsPlainText -Force)

Sleep 20

New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId

$sub = Get-AzureRmSubscription

$sub.TenantId
$sub.SubscriptionId

Now that you have the template and settings file you can run the following command to let Packer build your image:


packer build -var-file=packersettings.json windows.json

This is going to take some time. Packer needs to deploy a Key Vault, a new VM, run your scripts, power off the VM, capture an image from it and clean up the temporary resources. A simple image can easily take 20 minutes. The full VSTS image will take 7-8 hours.

If something goes wrong, you can debug your build by setting the PACKER_LOG environment variable to 1: $env:PACKER\LOG=1. If you want to inspect the temporary Packer VM in its intermediate state, add the following argument to your packer build command: -on-error=abort. Passing this argument stops Packer from deleting the VM, allowing you to remote desktop into it and see what went wrong.

After packer build finishes, you can use the Azure portal to view the image that Packer created.

Click on Create VM to enter the standard wizard to create a new Azure VM. The only difference is that instead of using one of the default VM images, you’re now using your own custom image. If those steps succeed, you have created your first custom Packer build image and a resulting VM!

Something that bit me the first time I tried this was that I forgot to add the sysprep step as the last step in my image build. If you mis that step, you will get an image and be able to create a VM, but it will never boot.

What’s next

In this post, you’ve been introduced to Packer, created your first custom image and build a new VM from that image. In the next post I’ll show you PowerShell scripts for building the image, creating the VM and installing the VSTS agent. I will then show you how to setup a CI/CD pipeline that automate these tasks for you.