File encoding: UTF-8 and UTF-8 BOM

Before packaging an extension, I parse the JSON manifest and update the version number. This allows me to set my version at packaging time and then use the same version when deploying to several different environments.

While parsing my manifest file, I suddenly ran into the following error:

Error occurred while updating tasks version: Error determining tasks manifest paths: SyntaxError: Unexpected token

Somehow, my code wasn’t able to open manifest file and parse the JSON in it. After reaching out to the product group I found out that this had to do with the encoding of my JSON manifest file. If the encoding is UTF-8, everything works as it’s supposed to do. My manifest had an encoding of UTF-8 BOM. BOM stands for Byte Order Mark. This is a ‘magic string’ that appears at the start of a file to signal to a reader of the file things like encoding and endianness (see Wikipedia for more information).

All I had to do was convert my JSON manifest file from UTF-8 BOM to UTF-8. The easiest way that I found was using Notepad++. After opening the manifest file in Notepad++, you can change the encoding of the file from the encoding menu and then save the file to disk.

Encoding in Notepad++Saving the file as UTF-8 fixed the error.

Typings for TypeScript

When using TypeScript, you will need TypeScript definition files to work with external libraries. A lot of those definition files are available on GitHub: DefinitelyTyped. At the time of writing, there are 1708 entries which is to much to show, even for GitHub.

Let’s say you want to work with jQuery. If you look at the DefinitelyTyped folder for jQuery, you’ll find a couple of files:

jQuery-DefinitelyTyped

The extension .d.ts signals that the file is a TypeScript Definition file. To use these files in your project, you can choose to download the Definition file and copy it to your project. But manually downloading and searching for files doesn’t sound like the best option.

Fortunately, there is an easier way: Typings.

Using Typings for TypeScript

One scenario where I used Typings is while developing the open source Folder Management extension that you can find at GitHub.

The Folder Management project uses Node Package Manager to download external libraries that it uses. If you look at the package.json file you can see which packages are used in this application:

{
 "name": "folder-management",
 "version": "1.0.0",
 "description": "Microsoft DevLabs Folder Management extension",
 "keywords": [
 "vsts",
 "tfs"
 ],
 "scripts": {
 "initdev": "typings install",
 "copyfiles": "copyfiles -f node_modules/vss-sdk/lib/VSS.SDK.js node_modules/jquery/dist/jquery.min.js scripts/lib",
 "setup": "npm run initdev && npm run copyfiles",
 "package": "tfx extension create --manifest-globs vss-extension.json",
 "publish": "tfx extension publish --token <token> --manifest-globs vss-extension.json"
 },
 "author": "ALM Rangers",
 "license": "MIT",
 "devDependencies": {
 "copyfiles": "^0.2.1",
 "grunt": "~0.4.5",
 "grunt-cli": "^1.1.0",
 "grunt-contrib-copy": "~0.8.2",
 "grunt-exec": "~0.4.6",
 "grunt-typescript": "*",
 "jquery": "^2.2.2",
 "requirejs": "2.1.22",
 "tfx-cli": "^0.3.19",
 "tsconfig-glob": "^0.4.0",
 "typescript": "^1.7.5",
 "typings": "^0.6.6",
 "vset": "^0.4.24",
 "vss-web-extension-sdk": "^1.96.1"
 }
}

The dependency we’re now interested in is typings. Adding this line to your package.json makes sure that the typings library is downloaded to your node_modules folder. If you then look at the script sections you see a line named initdev that runs a typings install command.
The Task Runner Explorer shows the custom tasks that I defined in package.json. The initdev task is the one we’re interested in.

Task Runner Explorer - NPM tasks

When you run this script, typings looks for a typings.json file and starts downloading the definition files that you need. The typings.json file for Folder Management looks like this:

{
 "dependencies": { },
 "devDependencies": { },
 "ambientDevDependencies": {
 "Q": "github:DefinitelyTyped/DefinitelyTyped/q/Q.d.ts#4de74cb527395c13ba20b438c3a7a419ad931f1c",
 "jquery": "github:DefinitelyTyped/DefinitelyTyped/jquery/jquery.d.ts#470954c4f427e0805a2d633636a7c6aa7170def8",
 "knockout": "github:DefinitelyTyped/DefinitelyTyped/knockout/knockout.d.ts#4de74cb527395c13ba20b438c3a7a419ad931f1c",
 "tfs": "github:microsoft/vss-web-extension-sdk/typings/tfs.d.ts",
 "vss": "github:microsoft/vss-web-extension-sdk/typings/vss.d.ts"
 }
}

You configure a name and a url to each definition file that you want to have. In this case, I’m downloading the files for Q, jQuery, Knockout and the TFS and VSS sdks. The easiest ways to get the URL, is navigate to the file in GitHub and use the ‘Copy path’ button.

If you have the typings dependency downloaded through NPM and you have the typings.json file, you’re ready to run the typings install script. Running this script will give you a new folder named typings. In this folder, there is a main.d.ts file that references all your downloaded definition files. All you have to do, is add a reference to main.d.ts from your TypeScript files and you’re done.

Solution Explorer showing typings

And that’s all there is. Feel free to download the code from GitHub and open the solution in Visual Studio. You’ll then automatically download the NPM packages and be able to use the Task Runner to run the initdev script to download the typings.

Feel free to leave a comment!

 

Modifying a VSIX with Node

In a previous post I showed how to use PowerShell to edit a VSIX file. Because of the cross platform capabilities of the release and build system of Visual Studio Team Services, I ported the code to Node. This allows you to run the script on any of the supported platforms where the Node based agent runs such as Linux and Mac.

To create and test the script I first installed the Node.js tools for Visual Studio. This allows you to create  a Node.js application such as a Console App or a Web app.

image

To create the VSIX editing script, I created a Console Application. With Visual Studio you can then run and debug the script. The final extension has three important files.

package.json

Package.json is the configuration file for the Node.js package manager: NPM. In your package.json file you list the dependencies that are required for your application. You can also list scripts that are then available from the Task Runner (if you install the NPM Task Runner extension). The package.json file for this application looks like this:

{
 "devDependencies": {
 "adm-zip": "^0.4.7",
 "archiver": "^0.21.0",
 "copyfiles": "^0.2.1",
 "mkdirp": "^0.5.1",
 "q": "^1.4.1",
 "rimraf": "^2.5.1",
 "temp": "^0.8.3",
 "tfx-cli": "^0.3.12",
 "tsconfig-glob": "^0.4.0",
 "typescript": "^1.7.5",
 "typings": "^0.6.6",
 "vsts-task-lib": "^0.5.10",
 "x2js": "^2.0.0",
 "xmldom": "^0.1.22"
},
 "scripts": {
 "initdev:npm": "npm install",
 "initdev:typings": "typings install",
 "initdev": "npm run initdev:npm  && npm run initdev:typings",
 "compile": "tsc --project .\\"
},
 "name": "extensiondependencies",
 "private": true,
 "version": "0.0.0"
}

First, there is a list of development dependencies. All these dependencies are downloaded by NPM and put in the node_modules subfolder in your project. The scripts section lists a couple of scripts to initialize NPM and Typings and to compile the TypeScript files. These scripts can be run through the Task Runner Explorer or from the command line.

image

typings.json

The typings.json file describes the TypeScript definition files thar you use in your project:

{
 "dependencies": { },
 "devDependencies": { },
 "ambientDevDependencies": {
 "adm-zip": "github:DefinitelyTyped/DefinitelyTyped/adm-zip/
            adm-zip.d.ts",
 "jasmine": "github:DefinitelyTyped/DefinitelyTyped/jasmine/
            jasmine.d.ts",
 "node": "github:DefinitelyTyped/DefinitelyTyped/node/node.d.ts
            #1c56e368e17bb28ca57577250624ca5bd561aa81",
 "Q": "github:DefinitelyTyped/DefinitelyTyped/q/Q.d.ts
            #aae1368c8ee377f6e9c59c2d6faf1acb3ece7e05",
 "vsts-task-lib": "github:Microsoft/vso-agent-tasks/definitions/
            vsts-task-lib.d.ts#releases/m94",
 "temp": "github:DefinitelyTyped/DefinitelyTyped/temp/temp.d.ts",
 }
}

As you can see, a definition has a name and a location where the defenition file can be found. A lot of definition files can be found in the GitHub repository DefinitelyTyped. The definition file for the vsts-task-lib is located in the Microsoft repository at GitHub. Running typings install from a command line downloads all the definition files and stores them in a typings folder in your project. A main.d.ts file is created which references all your definition files so that you only have to reference one single file from your code.
<image

app.ts

The actual script is contained in the app.ts file. This is the startup file for your Node Console Application. Since it’s a TypeScript file, I created a class named vsixEditor that takes care of the actual work:

/// <reference path="typings/main.d.ts" />
import AdmZip = require("adm-zip")
import temp = require('temp');
import fs = require('fs');
import path = require('path');
import Q = require("q");

class VSIXEditor {
 private zip: AdmZip;
 private outputPath: string;
 private edit: boolean = false;

 private versionNumber: string = null;
 private id: string = null;
 private publisher: string = null;

 constructor(input: string, output: string) {
 this.outputPath = output;
 this.zip = new AdmZip(input);
 }

 public startEdit() {
 if (this.edit) throw "Edit is already started";
 this.edit = true;
 }

 public endEdit() {
 this.validateEditMode();

 if (this.hasEdits()) {
 temp.track();

 temp.mkdir("visxeditor", (err, dirPath) => {
 if (err) throw err;

 this.zip.extractAllTo(dirPath, true);

 this.EditVsixManifest(dirPath)
 .then(() => {
 this.EditVsoManifest(dirPath).then(() => {
 var archiver = require('archiver');
 var output = fs.createWriteStream(this.outputPath);
 var archive = archiver('zip');

 output.on('close', function () {
 console.log(archive.pointer() + ' total bytes');
 console.log('archiver has been finalized and the output file descriptor has closed.');
 });

 archive.on('error', function (err) {
 throw err;
 });

 archive.pipe(output);

 archive.bulk([
 { expand: true, cwd: dirPath, src: ['**/*'] }
 ]);
 archive.finalize();
 });
 });


 });
 }
 }
 private EditVsoManifest(dirPath: string) {
 var deferred = Q.defer<boolean>();

 var vsoManifestPath = path.join(dirPath, 'extension.vsomanifest');
 fs.readFile(vsoManifestPath, 'utf8', (err, vsoManifestData) => {
 if (err) throw err;
 fs.writeFile(vsoManifestPath, vsoManifestData, () => {
 deferred.resolve(true)
 });
 });
 return deferred.promise;
 }

 private EditVsixManifest(dirPath: string) {
 var deferred = Q.defer<boolean>();
 var x2jsLib = require('x2js');
 var x2js = new x2jsLib();

 var vsixManifestPath = path.join(dirPath, 'extension.vsixmanifest');
 fs.readFile(vsixManifestPath, 'utf8', (err, vsixManifestData) => {
 if (err) throw err;

 var vsixmanifest = x2js.xml2js(vsixManifestData);
 var identity = vsixmanifest.PackageManifest.Metadata.Identity;
 if (this.versionNumber) identity._Version = this.versionNumber;
 if (this.id) identity._Id = this.id;
 if (this.publisher) identity._Publisher = this.publisher;

 vsixManifestData = x2js.js2xml(vsixmanifest);

 fs.writeFile(vsixManifestPath, vsixManifestData, () => {
 deferred.resolve(true)
 });
 deferred.resolve(true);
 });


 return deferred.promise;
 }

 private hasEdits(): boolean {
 return this.versionNumber != null
 }
 public EditVersion(version: string) {
 this.validateEditMode();
 this.versionNumber = version;
 }

 public EditId(id: string) {
 this.validateEditMode();
 this.id = id;
 }

 public EditPublisher(publisher: string) {
 this.validateEditMode();
 this.publisher = publisher;
 }

 private validateEditMode() {
 if (!this.edit) throw "Editing is not started";
 }
}

And this is how you would use the code:

var vsixEditor = new VSIXEditor("C:/temp/myvsix.vsix",
 "C:/temp/myvsixoutput.vsix");
 vsixEditor.startEdit();
 vsixEditor.EditVersion("1.0.0");
 vsixEditor.EditId("xxIDxx");
 vsixEditor.EditPublisher("xxPublisherxx");
 vsixEditor.endEdit();

Modifying a VSIX file before publishing

A software delivery pipeline starts with a build. A build creates the artifacts you need. You then use these artifacts to deploy to different environments while only changing the configuration data that’s specific for an environment.

When it comes to Visual Studio Team Services extensions, the artifact that gets created by your build is a VSIX package. A VSIX is a ZIP file that contains metadata on your extension and the actual files required for your extension. There is no official way to edit a VSIX file after it’s been created. But when deploying a VSIX to different environments I want to be able to change things like the Publisher ID and the Version number. This allows me to have a single package and change the configuration data on deployment.

Manipulating a VSIX with PowerShell

Fortunately, the .NET Framework has support for manipulating the files in a ZIP file. The following script takes a VSIX, opens it up and updates values before publishing it to VSTS.

[cmdletbinding()]
param(
 [string] [Parameter(Mandatory=$true)] $PathToVSIX,
 [string] [Parameter(Mandatory=$true)] $Token,
 [string] $IsPublicInput = "false",
 [string] $Version = $null,
 [string] $Publisher = $null,
 [string] $RemoveBaseUriInput = "true",
 [string] $ShareWith= $null
)
Set-StrictMode -Version 3

[bool]$IsPublic = [bool]::Parse($IsPublicInput)
[bool]$RemoveBaseUri = [bool]::Parse($RemoveBaseUriInput)

$file = Get-ChildItem $PathToVSIX -Filter *.vsix -Recurse | % { $_.FullName } | Select -First 1
Write-Verbose "Found VSIX Package $file"

try { $null = [IO.Compression.ZipFile] }
catch { [System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem') }

try { $fileZip = [System.IO.Compression.ZipFile]::Open( $file, 'Update' ) }
catch { throw "Another process has locked the '$file' file." }

$desiredFile = [System.IO.StreamReader]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsixmanifest' }).Open()
$text = $desiredFile.ReadToEnd()
[xml]$xml = $text
$desiredFile.Close()
$desiredFile.Dispose()

if ($Version)
{
 Write-Verbose "Updating Version to $Version"
 $xml.PackageManifest.MetaData.Identity.Version = $Version
}

if ($Publisher)
{
 Write-Verbose "Updating Publisher to $Publisher"
 $xml.PackageManifest.MetaData.Identity.Publisher = $Publisher 
}

if($IsPublic -eq $true)
{
 Write-Verbose "Setting GalleryFlag to Public"
 $xml.PackageManifest.MetaData.GalleryFlags = "Public"
}
else
{
 Write-Verbose "Setting GalleryFlag to Private"
 $xml.PackageManifest.MetaData.GalleryFlags = ""
}

$desiredFile = [System.IO.StreamWriter]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsixmanifest' }).Open()

$desiredFile.BaseStream.SetLength(0)
$desiredFile.Write($xml.InnerXml)
$desiredFile.Flush()
$desiredFile.Close()

$desiredFile = [System.IO.StreamReader]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsomanifest' }).Open()
$text = $desiredFile.ReadToEnd()
$desiredFile.Close()
$desiredFile.Dispose()

if ($RemoveBaseUri -eq $true)
{
 $text = (($text -split "`n") | ? {$_ -notmatch 'baseUri'}) -join "`n"
}

$desiredFile = [System.IO.StreamWriter]($fileZip.Entries | Where-Object { $_.FullName -match 'extension.vsomanifest' }).Open()

$desiredFile.BaseStream.SetLength(0)
$desiredFile.Write($text)
$desiredFile.Flush()
$desiredFile.Close()

$fileZip.Dispose()

if($ShareWith -ne $null)
{
 $ShareWith = "--share-with" + $ShareWith
}
else
{
 $ShareWith = ""
}

npm install -g tfx-cli
tfx extension publish --vsix "$File" --token $Token $ShareWith

You can use this PowerShell script and execute it as a step in your release definition to configure and publish your extension.

New Visual Studio Team Services Dashboard Widget

Microsoft released new capabilities for building your own dashboard widgets. The Widget SDK is now in public preview. One of the advantages of being an ALM Ranger is that we get early access to these new features to help the team test them and deliver guidance.

Together with Mathias Olausen I’ve been working on a widget that shows a countdown on your dashboard. You can configure things like the front and background color, title and of course the date that you want to count down to. We also created a second widget that automatically retrieves the end of your current iteration and uses that for the countdown.

On the official ALM Rangers blog we’ve published an article that describes some key points of the widget and how we’ve build it.

  1. Getting started with Widgets
  2. Developing the Widget
  3. Issues and resolving them
  4. Publishing Widgets

So please go and install our new widget and let us know what you think!