Build your own Hosted VSTS Agent Cloud: Part 4 – Customize

2018-03-13

If you have followed the previous 3 parts (part 1, part 2, part 3) you have a full CI/CD pipeline that allows you to deploy your own private agents based on the official Microsoft configuration for the VSTS Hosted Agent Pool. One of the steps was importing the GitHub repository containing the Packer configuration in your own VSTS account. This allows you to easily customize the script and make sure that your private agents fit your needs. In this post I’ll show you some customization I made to the Packer configuration.

This post is the fourth in a series of 4:

Customizing the current configuration

If you look at the Readme file stored in the repository, you can easily see which software is installed on the agent. And it’s a lot. The file vs2017-Server2016-Azure.json shows that Microsoft calls a PowerShell script to install each component and another script to validate that the installation succeeded.

For example, if you don’t need Wix just remove the following two snippets from the json file:

{
   "type": "powershell",
   "scripts":[
      "{{ template_dir }}/scripts/Installers/Vs2017/Install-Wix.ps1"
   ]
},
{
   "type": "powershell",
   "scripts":[
      "{{ template_dir }}/scripts/Installers/Vs2017/Validate-Wix.ps1"
   ]  
},

The same is true for things like Java or Go. Just remove the JSON snippet and Packer won’t install those components.

Caching Docker images

If you’re doing Docker development, you’ve probably noticed that downloading Docker images takes time. Images such as SQL Server on Linux or ASP.NET are quite big and no matter how fast your internet connection, downloading these takes time. Fortunately, Microsoft already added a script that helps you cache Docker images on your VM. Update-DockerImages.ps1 defines a DockerPull function and caches a couple of default images:


function DockerPull {
   Param ([string]$image)

   Write-Host Installing $image ...

   $j = Start-Job -ScriptBlock { docker pull $args[0] } -ArgumentList $image

   while ($j.JobStateInfo.state -ne "Completed" -And $j.JobStateInfo.state -ne "Failed") {
      Write-Host $j.JobStateInfo.state
      Start-Sleep 10
   }

   $results = Receive-Job -Job $j
   $results
}

DockerPull microsoft/windowsservercore
DockerPull microsoft/nanoserver
DockerPull microsoft/aspnetcore-build:1.0-2.0
DockerPull microsoft/aspnet
DockerPull microsoft/dotnet-framework

If you need another image that you want to use for building or testing your app, this is the place to add them. That way, they will be downloaded once and will be available on all your Agents.

Caching NPM packages

Another part where I want to add some extra caching is npm. Since you’re using private agents you will already benefit for not having to download all npm packages on each build but if you have a couple of popular packages, you can add them to the end of the Node install script:


npm install -g gulp-cli
npm install -g grunt-cli
npm install -g bower
npm install -g cordova

Adding and customizing your own scripts

And of course, you can also install completely new tooling by adding a script. Microsoft has to balance how often a tool is used with the amount of disk space they have available on the VM. For example, at the time of writing there is an issue on the GitHub repo asking for the SonarQube scanner. Microsoft answer is that you can install this as a part of your build which is true. But now that you’re running your own agent, it’s easy to add a simple script that installs the SonarQube scanner on your image and as a result on all your Agents.

Another thing you can take advantage of is having a faster update cadence. Microsoft has a huge responsibility making sure that the Hosted Pool works for millions of users. This also means they do some thorough testing which takes time. Currently, after a Visual Studio update is released, it still takes a couple of weeks before the Agent is updated. No one stops you modifying your script to install the newest VS update, run some quick test and start using it before Microsoft releases an official update.

Conclusion

In this series I showed you how to use Packer, VSTS, Azure and all the work Microsoft did to create a CI/CD pipeline for building and deploying your own VSTS Agents. I hope that this series also showed you some of the powerful capabilities of VSTS and Azure when it comes to automating deployments. You can find all the scripts I created on GitHub. If something doesn’t work for you, please post an issue or send a PR. For now, I hope this series was useful for you and helps you in running your own agents!