Tag Archives: python

Dynamic Toll Data: My First “Real” Alexa Skill

Posted on by
3

Summary

This project uses a combination of screen-scraping and an API to obtain current travel speeds and the current variable toll costs for Interstate 66 inside the beltway in northern Virginia, and provides the information to users via Amazon’s Alexa voice service. To try it out, first enable it (either just say “Enable sixty six tolls” or visit Sixty Six Tolls in the Amazon skills store). From then on, just say “Open sixty six tolls.”

Resources

The project uses Alexa’s voice service. The code is written in Python 3, using the Alexa Skills Kit SDK for Python. The code runs on AWS’ lambda service. It also makes (minimal) use of DynamoDB to store user-specific information. Travel times are scraped from the Virginia DOT’s (VDOT’s) 511 Virginia Traffic Information website. The real-time toll prices are obtained via an API to VDOT’s SmarterRoads data portal. Web scraping and XML parsing was done using Python’s Beautiful Soup library.

The python code as well as the interaction model (a JSON file) are available at https://github.com/ViennaMike/I-66-Tolls

Background

I was looking for a project that would use one of the data sets on the SmarterRoads portal, and I decided that it would be useful to be able to check the dynamic tolls on Interstate 66 inside the Beltway in Northern Virginia. Inbound traffic is tolled between 5:30 and 9:30 am, while outbound traffic is tolled between 3:00 and 7:00 pm.

The tolls are dynamically adjusted to maintain high speeds. While the toll may change between checking the cost at home and the time a driver reaches their entrance, it’s still useful to have an idea of the highly variable toll, especially since tolls for the entire 10 mile length have sometimes spiked at over $40. 

I had previously written a simple quiz skill using Amazon’s template, but this was my first custom skill.

Description

The overall architecture for the Alexa skill is shown below:

High Level Architecture

When a user interacts with the skill, the input is processed according to the interaction model that the developer defines in the Alexa Skill builder. This is captured in a JSON file. The skill builder is also where you tell the skill where to find the execution code for handling requests and ready a skill for certification and distribution.

In the case of 66 Tolls, there are eight custom intents, along with the Alexa built-in intents such as HelpIntent, FallbackIntent, StopIntent, etc. The custom intents are:

  • get_speeds for getting speeds and travel times for the two roughly parallel travel options (I-66 and US-50)
  • get_toll_hours to get static information on the hours that tolls are in effect
  • get_details to get additional static information about how the dynamic tolling system works
  • list_interchanges to get a list of inbound and outbound entrances and exits
  • get_toll to get the current toll price for a specified direction from a given entrance to exit
  • save_trip to save the user’s most frequent entrance and exit in each direction
  • get_favs to report back to the user what trips he has previously saved.
  • get_specific_help to provide help on particular types of requests (get toll, get speeds, and save trips).

When the skill is opened by a user who has previously saved a trip, the skill immediately returns the appropriate current inbound toll if it’s in the morning, or outbound toll if it’s in the evening or afternoon.

The Alexa Skills Kit SDK contains built-in functions to simplify interacting with Amazon’s DynamoDB NoSQL database. This skill uses a simple DynamoDB table to store the user_id (the key), most frequent inbound entrance and exit, and most frequent outbound entrance and exit.

By far the easiest part of the project was the code to get the travel times and tolls from the two VDOT sources. There is an API for the tolling data, and I had to do some simple web-scraping to get the travel time data. This code can be found in the get_travel_times() and get_tolls() functions in the code.

Developing the voice interaction model took multiple iterations, and I found that as time went on, I was able to improve the dialog model while reducing the number of intents and number of slot types associated with each intent. However, even then, I found that the first released version of my skill didn’t work for users the way I had intended. For the most part is worked well technically (there was one bad bug), but users other than myself said things differently than I had thought they would, and asked for help differently. It definitely pays off to not just spend time thinking of how users will interact with your skill (as I did at first), but in having others test out your skill as well, and getting there feedback.

Because this was new to me, it took considerable time, as well as trial and error, to figure out how to write the handler code, and especially how to handle the session and persistent attributes and the interaction with DynamoDB. I used a large number of resources, with the best being the Skill SDK documentation, the Color Picker sample app, and A Beginner’s Guide to the New AWS Python SDK for Alexa, by Ralu Bolovan. As described in the documentation, the python SDK supports two coding models, one based on functions with decorators, and the other based on classes. I chose to use classes, but the Color Picker example uses decorated functions.

Some of the hassles I had came from two factors: 1) the interface to Alexa skills has changed over time. It’s been improving, but this also meant that some of the samples and tutorials on the web are out of date. 2) While there is thorough documentation, a lot of tutorials and examples focus on simple demonstrations. For this reason, it may have been better for to step back and read more of the SDK rather than always jumping in. For example, I needed to have my code do something every time an intent was called, regardless of what the intent was. It turns out this is handled by Request Interceptors and Response Interceptors, which most of the simple samples leave out. This, along with the thorough walk-through on using DynamoDB, is why I found the Beginner’s Guide to the New AWS Python SDK for Alexa to be so helpful.

I originally wanted the invocation for the skill to be “i sixty six tolls,”, but I found that Alexa had trouble recognizing this as the invocation. For this reason, I had to make the invocation “sixty six tolls” rather than “i. sixty six tolls.”

I also found that if you use Alexa’s built in “confirm” capability, then when your code is called for the first time, the handler_input.request_envelope.session.new is set to False, apparently because the built-in confirmation request kicked off the session. That’s something to be careful of. For this and other reasons, I ended up checking whether or not I had previously initialized the session attributes, rather than whether or not the session was new.

The last technical bug I fixed was that I hadn’t thought about what “local time” was to the server. I have been naively thinking that since I was using AWS’s Northern Virginia server, the local time would be the US eastern time zone, but all Lambda servers use GMT as their local time, which makes a lot more sense. So I used the pytz library to convert to local time.

For the voice interface, I found I had to expand the list of synonyms for slot values (such as the names used for exits), add more specific help queries, besides the comprehensive, and therefore long, “help” intent, and make use of the interface’s built-in capabilities for checking user-provided slot values, which I hadn’t read about in the simple tutorials.

I hope that this example will prove helpful for others who want to write their own custom Alexa skills.

Calibration for 3-axis accelerometer and magnetometer

Posted on by
Reply

For my latest project, I’m using an 3-axis accelerometer / magnetometer, specifically Adafruit’s Triple-axis Accelerometer+Magnetometer (Compass) Board that uses the LSM303DLHC chip. .

The 3-Axis Magnetometer and Accelerometer Board

The 3-Axis Magnetometer and Accelerometer Board

 

My specific project may not require much precision, but I still decided it would be a good idea to calibrate the outputs. There are a number of  both sophisticated and simpler tools for magnetometer bias measurements, but I wanted a simple solution I could run on my headless Pi zero W and that dealt with calibrating both the magnetometer and accelerometer.  An excellent discussion that I borrowed heavily from is the blog post “How to Calibrate a Magnetometer.”

There are two general types of distortions in the outputs. One is a straight shift (bias) in a given direction, while the other distorts the response. The former is the easiest to deal with computationally. One can rotate the unit around all axes (a figure 8 movement with rotation is what I use) and measure the minimum and maximum values reported for the 3 axes (six values in all). If there is no bias, the absolute values for the + and – direction of each axis would be the same. If the response is shifted towards one or the other, there’s a bias present. One can simply compute the average of the min and max for each axis, and then subtract that value from the reported output for that axis.

plot showing an offset circle.

Offset bias. Called “hard iron distortion” in a magnetometer, as it is often due to the presence of exterior ferrous substances. (source: Vectornav)

The other type of error distorts the shape of the response. 3D matrix algebra is needed to fully address this type of error, but a simpler approach that provides an approximate correction is to just look at the 3 axes. Calculate the average delta between the maximum and minimums for each axis, then compute the scaling factor for each individual axis that scales the delta for that axis to match the average.

Plot showing rotated and offset ellipse

Plot showing both offset bias and non-linear response. The non-linear response is called “soft iron” bias in a magnetometer. (source: Vectornav)

My code then does the same thing for the accelerometer. For that calibration, you want to slowly (so as not to introduce large motion accelerations) position the board so that each of the 6 faces points up while the calibration program is running.

One that is done, the code then writes the magnetometer and accelerometer offsets and scaling factors for each axis (so 12 values in total) to a .ini file so that they can be called and used by the application program that will be making the measurements.

In my initial measurements, I found appreciable bias errors for the magnetometer (on the order of 12%), with much smaller bias errors for the accelerometer (on the order of 1.5%). The scaling factor corrections were  smaller for the magnetometer than the bias (the correction factors were 7.5%, 1%, and 6%). For the accelerometer, I measured 0.03%, 6%, and 5.5%).

The code is published at https://gist.github.com/ViennaMike/d8b8f9636694c7edf4f115b28c9378c0

Open Source + Free Tier Services: A Cornucopia for Hobbyists

Posted on by
Reply

Open Source logo I am constantly amazed by the amount, variety, and quality of open source software that is available, as well as the free (for small volume users) commercial Information and Communications Technology (ICT) services that exist. Open source has, in many domains, proven itself to be a reliable and cost-effective means of developing and improving software. I think an economic surplus for commercial companies has also contributed to this movement, as it has contributed to the free services that are available. (Although no doubt the free services are also intended as a gateway to the companies’ paid services.)

A little application I wrote the other evening provides an excellent example of this in action. The application is a software program that once a day checks inventory at our closest liquor stores for a certain hard to find and rarely in stock bourbon. If that particular bourbon is in stock at either store, it sends out a text alert. I wrote the application in Python, an open source programming language. It’s among the most popular programming languages in the world (and almost all its competitors are also open source). It used to be that companies paid large sums of money for commercial language compilers, while companies like Microsoft and Borland sold programming languages for PCs at hundreds of dollars per copy. Now programming language software is available for free, almost all as open source, while many software development environments are also free, and many open source.

The total volume of code involved in this application to open a web page, render it, scrape it for content, and send a text message with the result is huge, yet I only needed to write 34 lines of code! How? By using open source libraries that others had developed. These libraries (reusable modules with well-defined interfaces) did all the heavy lifting, I just needed to write some code to glue them together in a certain way and provide the key parameters. There are currently over 100,000 third party libraries available as free as open source in the Python Package Index repository. Cost to use any of these packages? $0.00. Similar libraries exist for other software languages, such as PhP, C, and Javascript.

As mentioned above, the program sends out a text message if the inventory is greater than zero at one of the local stores. How does it do this? Well, it uses the free twilio.rest library provided by Twilio. Twilio provides automated SMS (text), voice call, and other telecommunications services that you access through your own software programs. They are a commercial company that charges for their services, but you can test it out for free, and according to some folks, very low volume users don’t seem to ever exhaust their free account. This is an example where hobbyists like myself can, at no cost, take advantage of services developed for larger commercial customers.

At first, I ran this application on my PC at home, using the Windows Task Scheduler to have it automatically run each morning. But a) the Windows scheduler is both a pain and a bit flaky and b) it won’t run if my PC is powered off. So, I ported my application over to run “in the cloud” on Amazon Web Services, using what Amazon calls their Lambda service. As Amazon describes it, “AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running… upload your code and Lambda takes care of everything required to run and scale your code with high availability.”  As with Twilio, this is a commercial fee-for-service offering, except that you can use it for free if you call your services less than one million times per month and use less than 3.2 million seconds of compute time per month.

Finally, all those servers at Amazon need to run an operating system. What operating system do they use? Mostly Linux. While Windows and MacOS may dominate the PC world, the overwhelming majority of servers in the world run Linux. And like Python, Linux is free and open source. Linux is also at the heart of the Android operating system used on many phones and can be found running smart TVs and the infotainment systems on many cars.

Software is fundamentally different from hardware, so you can’t fully duplicate this environment for physical things (although there’s also a much smaller Open Hardware movement that open sources the design of hardware, rather than patenting it or keeping it a trade secret).  Nevertheless, it’s amazing how much can be done solely with the combination of open source software and free services!