File encoding: UTF-8 and UTF-8 BOM

Before packaging an extension, I parse the JSON manifest and update the version number. This allows me to set my version at packaging time and then use the same version when deploying to several different environments.

While parsing my manifest file, I suddenly ran into the following error:

Error occurred while updating tasks version: Error determining tasks manifest paths: SyntaxError: Unexpected token

Somehow, my code wasn’t able to open manifest file and parse the JSON in it. After reaching out to the product group I found out that this had to do with the encoding of my JSON manifest file. If the encoding is UTF-8, everything works as it’s supposed to do. My manifest had an encoding of UTF-8 BOM. BOM stands for Byte Order Mark. This is a ‘magic string’ that appears at the start of a file to signal to a reader of the file things like encoding and endianness (see Wikipedia for more information).

All I had to do was convert my JSON manifest file from UTF-8 BOM to UTF-8. The easiest way that I found was using Notepad++. After opening the manifest file in Notepad++, you can change the encoding of the file from the encoding menu and then save the file to disk.

Encoding in Notepad++Saving the file as UTF-8 fixed the error.

My VSTS Presentation at SDN

Last week I had the privilege of giving a presentation at the Software Development Network conference. There was a nice turn-out of around 50 people. After a couple of slides I had one big demo that showed the different elements of Visual Studio Team Services such as Plan, Code, Build, Test, Release and Monitor.

You can find the presentation here: From 0 to 60. After the slides I included a series of screenshots that take you through the demo (in case of a failing internet connection!).

During the presentation we had some interesting questions and discussion points like:

  • My team uses an on-premises TFS. When will all the new features of VSTS be available for TFS?

You can check the Visual Studio Team Services Features Timeline to see when features will be available for TFS.

  • My testers work in an Agile way. Can they use the testing features without having to create huge test cases upfront?

Yes. You can use Exploratory Testing. You can run an Exploratory Testing session with Microsoft Test Manager or with the new extension for Chrome.

  • What’s Kanban?

Through a demo I showed how to use and configure the Kanban board. To get a grasp of the basics of Kanban, you can find info on MSDN.

  • What’s the Marketplace for VSTS and is it useful?

It’s absolutely useful! The Marketplace allows you to find an install extensions for Visual Studio, Visual Studio Team Services, Team Foundation Server and Visual Studio Code. You can find small, easy to use extensions like Folder Management and Branch Delete and more complex extensions like Work Item Visualization and File owner.

  • Who are the ALM Rangers?

I always insert a slide that explains what we do as ALM Rangers. Often people haven’t heard of us yet but have already used guidance or tooling that we build. You can find more info at aka.ms/vsarunderstand.

Questions or other feedback? Please leave a comment. 

 

Modifying a VSIX with Node

In a previous post I showed how to use PowerShell to edit a VSIX file. Because of the cross platform capabilities of the release and build system of Visual Studio Team Services, I ported the code to Node. This allows you to run the script on any of the supported platforms where the Node based agent runs such as Linux and Mac.

To create and test the script I first installed the Node.js tools for Visual Studio. This allows you to create  a Node.js application such as a Console App or a Web app.

image

To create the VSIX editing script, I created a Console Application. With Visual Studio you can then run and debug the script. The final extension has three important files.

package.json

Package.json is the configuration file for the Node.js package manager: NPM. In your package.json file you list the dependencies that are required for your application. You can also list scripts that are then available from the Task Runner (if you install the NPM Task Runner extension). The package.json file for this application looks like this:

{
 "devDependencies": {
 "adm-zip": "^0.4.7",
 "archiver": "^0.21.0",
 "copyfiles": "^0.2.1",
 "mkdirp": "^0.5.1",
 "q": "^1.4.1",
 "rimraf": "^2.5.1",
 "temp": "^0.8.3",
 "tfx-cli": "^0.3.12",
 "tsconfig-glob": "^0.4.0",
 "typescript": "^1.7.5",
 "typings": "^0.6.6",
 "vsts-task-lib": "^0.5.10",
 "x2js": "^2.0.0",
 "xmldom": "^0.1.22"
},
 "scripts": {
 "initdev:npm": "npm install",
 "initdev:typings": "typings install",
 "initdev": "npm run initdev:npm  && npm run initdev:typings",
 "compile": "tsc --project .\\"
},
 "name": "extensiondependencies",
 "private": true,
 "version": "0.0.0"
}

First, there is a list of development dependencies. All these dependencies are downloaded by NPM and put in the node_modules subfolder in your project. The scripts section lists a couple of scripts to initialize NPM and Typings and to compile the TypeScript files. These scripts can be run through the Task Runner Explorer or from the command line.

image

typings.json

The typings.json file describes the TypeScript definition files thar you use in your project:

{
 "dependencies": { },
 "devDependencies": { },
 "ambientDevDependencies": {
 "adm-zip": "github:DefinitelyTyped/DefinitelyTyped/adm-zip/
            adm-zip.d.ts",
 "jasmine": "github:DefinitelyTyped/DefinitelyTyped/jasmine/
            jasmine.d.ts",
 "node": "github:DefinitelyTyped/DefinitelyTyped/node/node.d.ts
            #1c56e368e17bb28ca57577250624ca5bd561aa81",
 "Q": "github:DefinitelyTyped/DefinitelyTyped/q/Q.d.ts
            #aae1368c8ee377f6e9c59c2d6faf1acb3ece7e05",
 "vsts-task-lib": "github:Microsoft/vso-agent-tasks/definitions/
            vsts-task-lib.d.ts#releases/m94",
 "temp": "github:DefinitelyTyped/DefinitelyTyped/temp/temp.d.ts",
 }
}

As you can see, a definition has a name and a location where the defenition file can be found. A lot of definition files can be found in the GitHub repository DefinitelyTyped. The definition file for the vsts-task-lib is located in the Microsoft repository at GitHub. Running typings install from a command line downloads all the definition files and stores them in a typings folder in your project. A main.d.ts file is created which references all your definition files so that you only have to reference one single file from your code.
<image

app.ts

The actual script is contained in the app.ts file. This is the startup file for your Node Console Application. Since it’s a TypeScript file, I created a class named vsixEditor that takes care of the actual work:

/// <reference path="typings/main.d.ts" />
import AdmZip = require("adm-zip")
import temp = require('temp');
import fs = require('fs');
import path = require('path');
import Q = require("q");

class VSIXEditor {
 private zip: AdmZip;
 private outputPath: string;
 private edit: boolean = false;

 private versionNumber: string = null;
 private id: string = null;
 private publisher: string = null;

 constructor(input: string, output: string) {
 this.outputPath = output;
 this.zip = new AdmZip(input);
 }

 public startEdit() {
 if (this.edit) throw "Edit is already started";
 this.edit = true;
 }

 public endEdit() {
 this.validateEditMode();

 if (this.hasEdits()) {
 temp.track();

 temp.mkdir("visxeditor", (err, dirPath) => {
 if (err) throw err;

 this.zip.extractAllTo(dirPath, true);

 this.EditVsixManifest(dirPath)
 .then(() => {
 this.EditVsoManifest(dirPath).then(() => {
 var archiver = require('archiver');
 var output = fs.createWriteStream(this.outputPath);
 var archive = archiver('zip');

 output.on('close', function () {
 console.log(archive.pointer() + ' total bytes');
 console.log('archiver has been finalized and the output file descriptor has closed.');
 });

 archive.on('error', function (err) {
 throw err;
 });

 archive.pipe(output);

 archive.bulk([
 { expand: true, cwd: dirPath, src: ['**/*'] }
 ]);
 archive.finalize();
 });
 });


 });
 }
 }
 private EditVsoManifest(dirPath: string) {
 var deferred = Q.defer<boolean>();

 var vsoManifestPath = path.join(dirPath, 'extension.vsomanifest');
 fs.readFile(vsoManifestPath, 'utf8', (err, vsoManifestData) => {
 if (err) throw err;
 fs.writeFile(vsoManifestPath, vsoManifestData, () => {
 deferred.resolve(true)
 });
 });
 return deferred.promise;
 }

 private EditVsixManifest(dirPath: string) {
 var deferred = Q.defer<boolean>();
 var x2jsLib = require('x2js');
 var x2js = new x2jsLib();

 var vsixManifestPath = path.join(dirPath, 'extension.vsixmanifest');
 fs.readFile(vsixManifestPath, 'utf8', (err, vsixManifestData) => {
 if (err) throw err;

 var vsixmanifest = x2js.xml2js(vsixManifestData);
 var identity = vsixmanifest.PackageManifest.Metadata.Identity;
 if (this.versionNumber) identity._Version = this.versionNumber;
 if (this.id) identity._Id = this.id;
 if (this.publisher) identity._Publisher = this.publisher;

 vsixManifestData = x2js.js2xml(vsixmanifest);

 fs.writeFile(vsixManifestPath, vsixManifestData, () => {
 deferred.resolve(true)
 });
 deferred.resolve(true);
 });


 return deferred.promise;
 }

 private hasEdits(): boolean {
 return this.versionNumber != null
 }
 public EditVersion(version: string) {
 this.validateEditMode();
 this.versionNumber = version;
 }

 public EditId(id: string) {
 this.validateEditMode();
 this.id = id;
 }

 public EditPublisher(publisher: string) {
 this.validateEditMode();
 this.publisher = publisher;
 }

 private validateEditMode() {
 if (!this.edit) throw "Editing is not started";
 }
}

And this is how you would use the code:

var vsixEditor = new VSIXEditor("C:/temp/myvsix.vsix",
 "C:/temp/myvsixoutput.vsix");
 vsixEditor.startEdit();
 vsixEditor.EditVersion("1.0.0");
 vsixEditor.EditId("xxIDxx");
 vsixEditor.EditPublisher("xxPublisherxx");
 vsixEditor.endEdit();

Modifying a VSIX file before publishing

A software delivery pipeline starts with a build. A build creates the artifacts you need. You then use these artifacts to deploy to different environments while only changing the configuration data that’s specific for an environment.

When it comes to Visual Studio Team Services extensions, the artifact that gets created by your build is a VSIX package. A VSIX is a ZIP file that contains metadata on your extension and the actual files required for your extension. There is no official way to edit a VSIX file after it’s been created. But when deploying a VSIX to different environments I want to be able to change things like the Publisher ID and the Version number. This allows me to have a single package and change the configuration data on deployment.

Manipulating a VSIX with PowerShell

Fortunately, the .NET Framework has support for manipulating the files in a ZIP file. The following script takes a VSIX, opens it up and updates values before publishing it to VSTS.

[cmdletbinding()]
param(
 [string] [Parameter(Mandatory=$true)] $PathToVSIX,
 [string] [Parameter(Mandatory=$true)] $Token,
 [string] $IsPublicInput = "false",
 [string] $Version = $null,
 [string] $Publisher = $null,
 [string] $RemoveBaseUriInput = "true",
 [string] $ShareWith= $null
)
Set-StrictMode -Version 3

[bool]$IsPublic = [bool]::Parse($IsPublicInput)
[bool]$RemoveBaseUri = [bool]::Parse($RemoveBaseUriInput)

$file = Get-ChildItem $PathToVSIX -Filter *.vsix -Recurse | % { $_.FullName } | Select -First 1
Write-Verbose "Found VSIX Package $file"

try { $null = [IO.Compression.ZipFile] }
catch { [System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem') }

try { $fileZip = [System.IO.Compression.ZipFile]::Open( $file, 'Update' ) }
catch { throw "Another process has locked the '$file' file." }

$desiredFile = [System.IO.StreamReader]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsixmanifest' }).Open()
$text = $desiredFile.ReadToEnd()
[xml]$xml = $text
$desiredFile.Close()
$desiredFile.Dispose()

if ($Version)
{
 Write-Verbose "Updating Version to $Version"
 $xml.PackageManifest.MetaData.Identity.Version = $Version
}

if ($Publisher)
{
 Write-Verbose "Updating Publisher to $Publisher"
 $xml.PackageManifest.MetaData.Identity.Publisher = $Publisher 
}

if($IsPublic -eq $true)
{
 Write-Verbose "Setting GalleryFlag to Public"
 $xml.PackageManifest.MetaData.GalleryFlags = "Public"
}
else
{
 Write-Verbose "Setting GalleryFlag to Private"
 $xml.PackageManifest.MetaData.GalleryFlags = ""
}

$desiredFile = [System.IO.StreamWriter]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsixmanifest' }).Open()

$desiredFile.BaseStream.SetLength(0)
$desiredFile.Write($xml.InnerXml)
$desiredFile.Flush()
$desiredFile.Close()

$desiredFile = [System.IO.StreamReader]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsomanifest' }).Open()
$text = $desiredFile.ReadToEnd()
$desiredFile.Close()
$desiredFile.Dispose()

if ($RemoveBaseUri -eq $true)
{
 $text = (($text -split "`n") | ? {$_ -notmatch 'baseUri'}) -join "`n"
}

$desiredFile = [System.IO.StreamWriter]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsomanifest' }).Open()

$desiredFile.BaseStream.SetLength(0)
$desiredFile.Write($text)
$desiredFile.Flush()
$desiredFile.Close()

$fileZip.Dispose()

if($ShareWith -ne $null)
{
 $ShareWith = "--share-with" + $ShareWith
}
else
{
 $ShareWith = ""
}

npm install -g tfx-cli
tfx extension publish --vsix "$File" --token $Token $ShareWith

You can use this PowerShell script and execute it as a step in your release definition to configure and publish your extension.

New Visual Studio Team Services Dashboard Widget

Microsoft released new capabilities for building your own dashboard widgets. The Widget SDK is now in public preview. One of the advantages of being an ALM Ranger is that we get early access to these new features to help the team test them and deliver guidance.

Together with Mathias Olausen I’ve been working on a widget that shows a countdown on your dashboard. You can configure things like the front and background color, title and of course the date that you want to count down to. We also created a second widget that automatically retrieves the end of your current iteration and uses that for the countdown.

On the official ALM Rangers blog we’ve published an article that describes some key points of the widget and how we’ve build it.

  1. Getting started with Widgets
  2. Developing the Widget
  3. Issues and resolving them
  4. Publishing Widgets

So please go and install our new widget and let us know what you think!