Guest post by Terren Peterson: Vice President, Platform Engineering @ Capital One

Originally posted here.

Background:

One of my goals for 2017 is to not just write more Alexa Skills, but make sure what I’m creating is of top quality. Part of the challenge is how to add instrumentation to the application, so you can analyze how actual users are interacting with the skill. I’ve tried out different methods, including writing out to DynamoDB tables and then analyzing in Excel. Fortunately the ecosystem is growing around Voice platforms, and VoiceLabs has a free Analytics product.

TL;DR:

VoiceInsights is a great measurement tool to record actual user data, and a huge step up from what is available from the Alexa console. It’s just a few extra lines of code and FREE!

 

Step 1 – Sign-up for VoiceLabs account

Start by signing up for a free account at VoiceLabs.co. It’s minimal information to get started, and doesn’t require a credit card or other fees.


VoiceLabs Open an Account

Account Setup

Then setup an application within the console.

VoiceLabs Setup Application

Application Setup

This will give you the API Key required to call the service from your skill.

Step 2 – Create a Deployment Pipeline for Lambda

If you typically use the browser-based IDE for Lambda, you’re going to need to spend some time setting up a build pipeline to include the Voice Insight SDK into your skill. If you already have this, go ahead and skip this step.

First, start by deploying a new EC2 instance with Bitnami nodejs AMI through the console or API’s. Here’s the specific AMI you will need:

Screen Shot 2017-02-02 at 9.33.15 AM

The AMI doesn’t include PIP, so you’ll also need that step.

Screen Shot 2017-02-02 at 9.33.20 AM

Next go ahead and install the AWS CLI to the instance by first running the following command.

Screen Shot 2017-02-02 at 9.33.27 AM

Once that is done, go into your AWS account and using IAM, create a user account that has full access to the S3 service.

Add User AWS

Go through the four steps, and once completed it will give you an access key ID and secret access key. Record those, and from the command line, configure the AWS CLI using them.

Screen Shot 2017-02-02 at 9.33.35 AM

That’s it, and you can now reuse this process or the instance itself later when needed.

Step 3 – Configure your skill to use Voice Analytics

First, on the build server created in the previous step, use npm to install the VoiceInsights SDK through the following command.

Screen Shot 2017-02-02 at 9.33.44 AM

Then, within your skill, include a reference to it, along with the application key that was generated in Step 1.

Screen Shot 2017-02-02 at 9.33.50 AM

Next, find where the session initiates in your skill, and add the following statement.

Screen Shot 2017-02-02 at 9.33.59 AM

Finally, in each intent you want to record activity, add the following logic.

Screen Shot 2017-02-02 at 9.34.06 AM

More examples are given in in the VoiceLabs console for reference, including different options on the attributes that are passed into the call, as well as how this works in other languages like Java and Ruby.

Attached to the bottom of this story is a GitHub repo that shows all of this in context of the code for reference.

Step 4 – Deploy to a Staging Area

Now we need to take the package from the EC2 instance, and put it someplace that can get picked up by the Lambda function. The easiest way is to create a package, then stage it to the data object service in AWS called S3. If you don’t already have an S3 bucket to use, go ahead in the console and create one. Here’s a screenshot in the AWS console for reference – just pick a unique name, and an AWS Region to deploy it into.

Screenshot of AWS Bucket

Screenshot of AWS S3 Console

First, make sure that you have the code for your skill saved locally on the build server. I’m using the file name “lambda.js” for reference. It should be in the same directory that we installed the VoiceInsights SDK from the prior step. When you’re ready to create the package, just create a zip file with all of the necessary files.

Screen Shot 2017-02-02 at 9.34.15 AM

We’re including the “-r” option in the zip command so that it picks up the dependent node libraries, including the voice-labs one that we installed in Step 3. That gives us a build package — then execute this CLI command to push the package out to an S3 bucket.

Screen Shot 2017-02-02 at 9.34.20 AM

This acts like a local Linux copy command that can then push out the zip file to the S3 bucket. If you are having problems with this step, check the credentials that were set up in Step 2 as they might not have full access to S3.

Step 5 – Activate Staging Environment

Within the lambda options for deploying the skill, choose “Upload a file from Amazon S3.” You can also do this within the CLI if you want to fully automate the pipeline, but I’d rather have this manual for now so I don’t inadvertently push a new release out.

Activate Staging Environment AWSScreenshot of the AWS Console for the Lambda Service

Once this step is complete, your build package replaces whatever previously was in the browser-based IDE.

Step 6 – Analyze Results

Once you’ve integrated this into the skill, the analytics engine from VoiceLabs processes the data. There are a couple of different useful views within the product, starting with a trend over time of the broad usage of the skill. This differs from what’s in the Amazon console as it provides it in real-time (the Amazon console normally has a 24-48 hour lag) and it breaks out sessions vs. users.

VoiceLabs Analyze ResultsScreenshot from the VoiceInsights console

There are also more in-depth views of the skill usage, going into how what intent is being used the most, and how long the sessions are held for. No real equivalent to this in the Amazon console.

New vs Returning - VoiceLabsScreenshot from the VoiceInsights console

If you’re looking for how each user has used the skill, that’s all tracked. The VoiceInsights tool creates a proxy user account for each that matches what the device is passing in (sort of like a MAC address for the account).

Top Users VoiceLabsScreenshot from the VoiceInsights console

There’s also a way to track the flow of the session, and see how different paths are being taken.

VoiceLabs Voice PathingScreenshot from the VoiceInsights console

Conclusion

VoiceInsights is a great measurement tool to record actual user data, and a huge step up from what is available from the console. It’s just a few extra lines of code and FREE!