Gathering California Sales Tax Rates with Node.js Part 1: Downloading

In this post, I will discuss how I used Node.js to gather the California sales tax rates for display on my sales tax reference site http://www.sale-tax.com.  California is a good starting point since the data is available in an easy to read csv file on the ca.gov website: http://www.boe.ca.gov/sutax/files/city_rates.csv

Downloading the CSV

The first step in getting sales tax rates for California is to download the .csv file containing the rates.  The Node.js http.get(options, callback) method is used to initiate an http GET request.  Unfortunately, this method does not directly return the data to the callback so we will add a wrapper method that does that for us to make it easier for us to perform this common operation.  Let’s start by building up the options parameter from a url host and path and throwing an exception if we hit any error:

var http = require('http'); // pull in the http module

var httpGet = function(host, path, callback) {
  // fill in the options for the specified path
  var options = {
    host: host, // 'www.example.com'
    path: path, // '/path/to/file.txt'
    port: 80
  };
  http.get(options, function(res) {
    // TODO: retrieve data from res
  }).on('error', function(e) {
    throw Error('Error getting http://' + host + path + ' ' + e);
  });
};

Getting the Data

The next thing to do is use the http.ClientResponse object we receive in the http.get callback to download the contents of the csv. Inside the callback, we first initialize a local variable called data that will store the file as it is downloaded.

  ...
  http.get(options, function(res) {
    var data = '';

We need to set the encoding on res to 'utf8' so the data it gives us is in the form of  a JavaScript String instead of a Node.js Buffer.

    res.setEncoding('utf8');

Now we need to respond to the ClientResponse 'data' event which has a chunk of the downloaded file. Since we set the encoding to 'utf8', our event handler will retrieve a String that we can concat on the end of our data local variable:

    res.on('data', function (chunk) {
      // Append the chunk to the data downloaded so far
      data += chunk;
    });

We need to hook into the ClientResponse 'end' event to know when the entire file has downloaded.  When we get the 'end' event, data will contain the whole file and can pass it to the original callback to let the caller continue execution:

    res.on('end', function(){
      // All done downloading. Give the data to the caller.
      callback(data);
    });

Trying it out

Let’s put it all together with a simple test to verify it is working

var http = require('http');

var httpGet = function(host, path, callback) {
  // fill in the options for the specified path
  var options = {
    host: host, // 'www.example.com'
    path: path, // '/path/to/file.txt'
    port: 80
  }
  http.get(options, function(res) {
    var data = '';
    res.setEncoding('utf8');
    res.on('data', function (chunk) {
      // Append the chunk to the data downloaded so far
      data += chunk;
    });
    res.on('end', function(){
      // All done downloading. Give the data to the caller.
      callback(data);
    });
  }).on('error', function(e) {
    throw Error('Error getting http://' + host + path + ' ' + e);
  });
};

httpGet('www.boe.ca.gov','/sutax/files/city_rates.csv', function(data) {
  console.log(data);
});

Running this produces a long string of text to the console that ends with the contents of our file:

...
Yucca Valley*,7.750%,San Bernardino
Zamora,7.250%,Yolo
Zenia,7.250%,Trinity
,,
"Rates effective July 1, 2012",,
"""*"" next to city indicates incorporated city",,

We did it!

Next Steps

In our next post, we will process the downloaded csv file to build up the data for display on the California sales tax rate table.

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: