Managing e-statements for fun and profit

Unless you’ve been living under a rock for the last 5 years, you should have noticed the imminent extinction of paper statements from most of the consumer services. Whether we like it or not, e-bills are the way of the future.

One problem with e-bills is that they still need to be stored – preferably, in a way that makes them easy to find. One also cannot rely on the service owners to archive old bills, as they rarely keep them for longer than 1 year.

A typical process managing such bills involves multiple steps:

  • Download the new bill from the website of the service provider.
  • Rename the PDF file from an exciting automatic (e.g 8B96766CBC0A49E6A45C2B039623C56C.pdf) to a boring one (e.g. TRowePrice-2017-Q3.pdf)
  • Store it so that one could find it when needed (ideally, not in a single folder holding thousands of such files).

About 3 years ago, I’ve written a relatively simple tool that attempts to automate the latter two of those 3 steps (automatically downloading is notoriously difficult, given different website organization for each provider and the fact that they love to tweak the website layout, frequently breaking the scraping procedure).

The tool is called classify_bills and is available on GitHub. When run on a bunch on PDF files, it tries to figure out what account each file belongs to and what time period it is for. It then renames it appropriately and moves it into an appropriate destination.

The tool is configuration-driven, it basically runs off a bunch of regular expressions (hat tip to jwz), and configuring a new account is strictly speaking, not pleasant. Still, once configured, it eliminates the chore of laboriously doing those actions manually, bill by bill, month by month, making the process faster and pleasanter.

Posted in Development, Uncategorized | Tagged | Comments Off on Managing e-statements for fun and profit

Using Raspberry Pi to monitor and water plants


I’ve been growing hot peppers on my balcony and one of the things I constantly fail to keep on schedule is watering them. As it is with many chores, this is something simple yet not exciting enough to avoid being actively procrastinated on.

As such, I only saw several solutions to that problem:

  • Become better at avoiding procrastination (yeah, ri-i-i-ight!)
  • Outsource the activity to some hapless family member.
  • Get rid of the peppers.
  • Throw technology (and time, and money) at the problem.

Given all those options, the choice was rather simple. As a result, I built a computer-driven watering station that takes care of my peppers for me. At this point, my involvement is only needed to periodically add water to the water tank (about once every 2-3 weeks). Apart from that, the system is fully self-driven.

In its current incarnation, the system handles 4 pots (number of pots is limited by current hardware – in theory it should be possible to add more relays and moisture sensors). It periodically measures moisture level of the soil in each of the pots. When it decides the soil is “dry enough” it waters the plant. It also makes sure watering has the desired effect (i.e. preventing conditions like tank running empty or watering tube being misplaced). Finally, it posts all kinds of stats on a dashboard. (not yet, actually).

Bill of materials

Principles of operation

The idea is to use off-the-shelf components rather than more complex pre-assembled pieces.

The role of the controller is played by Raspberry Pi. In theory, the device could be similarly implemented using a simpler board, like Arduino, but Raspberry Pi is a full-fledged computer and is very flexible as such, so I decided to trade some simplicity and power economy for potential future flexibility.

The computer takes care of the following hardware:

  • 4 pumps, which it operates via relay (conveniently enough, a 4-relay board is available).
  • 4 analog moisture sensors (read through ADC as Pi doesn’t have analog GPIO).

In addition to that, I also wanted a convenient way to cleanly shut the device down, as opposed to just yanking the power cord (might corrupt the filesystem) or ssh’ing into the device (inconvenient). For that purpose, I’ve added a dedicated button – pressing it for 5 seconds initiates a clean shutdown.


(This is my first attempt at Fritzing, so pardon the crow’s nest of the wires…)

Building the system

One great idea I borrowed from here was to use USB charger as a power source for the pumps. Taking it a bit further, I’ve also used the same charger to feed the Raspberry Pi, which resulted in a fairly neat design.

Having a solid enclosure is critical as the device is going to be outdoors, subject to elements. Luckily, there are plenty of outdoor-rated electrical equipment boxes, which could be utilized for that purpose.

One final note is the temperature handling. The device is almost fully sealed (as it is located outside and is subject to elements) which might be an issue for Raspberry temperature. However, the load is light enough and so far the temps have been reasonable.


The system is managed using 2 scripts (see Github repo for details):

  • watering_station is responsible for monitoring and watering. It runs hourly via cron.
  • shutdown_button takes care of graceful shutdown when push button is pressed and held for 5 seconds. It starts at system boot via systemd and keeps running.

When setting up a new system, one has to make sure to enable I2C via raspi-config, otherwise connection to ADC will not work.

Future work

One of the future enhancements to the system will be adding a visual indicator of the activity. In the past, I periodically encountered situations where Raspberry Pi (previous generations, using external Edimax wireless adapter) would quietly lose the connection or sometimes get locked up after long uptime. Having a visual indicator of the device status would definitely help. That will be either an RGB LED (indicating status by different colors) or a tiny LCD display – I am yet to decide which one makes more sense.

Furthermore, since the device is already out there, I’m considering adding a Raspberry camera module to also show me the plants on demand. 🙂


Posted in Development, Smarthome | Tagged , , | Comments Off on Using Raspberry Pi to monitor and water plants

Rotating between different fonts in Emacs

There are two types of people: those who set on one font/color theme for the rest of their life and those like me, who constantly tweak their setup. Not that it helps productivity, but hey – RMS created .emacs for a reason!

Since typing in font names over and over again is a pain, I’ve created a neat function that can be called interactively to rotate between various programming-friendly fonts installed in the system. The list of fonts is partly mine, partly borrowed from Programming Fonts  (which is awesome on its own). This allows a very quick way to see how one’s code looks with different fonts (and make a final decision for the next 3 weeks or so).

(require 'cl)                           ; required for lexical-let

(lexical-let ((current-index 0))
  (defun switch-font ()
    "Rotate between programming-friendly fonts.

This function attempts to identify most of programming-friendly (mostly
monospace) fonts in the system. Each subsequent run of it switches the current
frame to the next available font allowing quick assessment of different fonts.

    (interactive "")
    (letrec ((font-list '("3270"
                          "Anonymous Pro"
                          "Aurulent Sans Mono"
                          "Average Mono"
                          "Bitstream Vera Sans Mono"
                          "Code New Roman"
                          "Courier New"
                          "Cutive Mono"
                          "DejaVu Mono"
                          "DejaVu Sans Mono"
                          "Droid Sans Mono"
                          "Effects Eighty"
                          "Fantasque Sans Mono"
                          "Fira Code"
                          "Fira Mono"
                          "Fixedsys with Ligatures"
                          "GNU Freefont"
                          "GNU Unifont"
                          "Generic Mono"
                          "Latin Modern Mono"
                          "Liberation Mono"
                          "Luxi Mono"
                          "Nova Mono"
                          "Office Code Pro"
                          "Oxygen Mono"
                          "PT Mono"
                          "Proggy Clean"
                          "Roboto Mono"
                          "SK Modernist Mono"
                          "Share Tech Mono"
                          "Source Code Pro"
                          "TeX Gyre Cursor"
                          "Ubuntu Mono"
                          "Verily Serif Mono"
             (valid-fonts (cl-intersection font-list (font-family-list)
                                           :test 'string=)))
        (setq current-index (mod (1+ current-index) (length valid-fonts)))
        (let ((font (nth current-index valid-fonts)))
          (set-frame-font font)
          (message (concat "Font switched to " font)))))))
Posted in Development | Tagged , | Comments Off on Rotating between different fonts in Emacs

Translit длѩ Яꙁꙑка Боꙗрскѧго

Первое практическое применение JavaScript на пользу человечеству (после динамических меню): транслит на Язык Боярский!







Posted in Uncategorized | Tagged , | Comments Off on Translit длѩ Яꙁꙑка Боꙗрскѧго

Creating reproducible ZIP archives

An Artist A DayI am currently working on an app that loads data in relatively small incremental payloads from the server side for offline use. Payload files are in ZIP format and the way the app knows whether it has the correct file (which might not be true – the file might be missing, partially downloaded or simply changed by developer) is by checking it’s checksum against the local file. If there is a checksum mismatch, the file is (re)downloaded.

During development, I regenerate the payload files many times and when the time comes, I release the “golden” versions into production. Obviously, I insist on reproducible builds and this includes the bundle files – if the bits of the data that went into those files haven’t changed, there should be absolutely no reason why the generated bundle should differ in a single bit. After all if the file has a new signature, the app will download it again (even if the data is the same). I could devise a more elaborate way via some kind of versioning but it would be more complex, more error-prone, and wouldn’t add much benefit to the user. Plus, it’s nice to know that your data is guaranteed to be the same.

Image credit: New Line Cinema.

Image credit: New Line Cinema.

Turns out, things are not so easy when it comes to ZIP files. Both OS X and Ubuntu offer command-line zip program which by default sticks gobs of metadata into the ZIP archive, including timestamps. Thus, no two zips are alike (well, maybe they will be if run within the same second, but I didn’t bother to check).

Unfortunately there is no way to tell zip to only include most critical metadata (file names and sizes) and ignore everything else. It does offer -X (aka -no-extra) option to strip some attributes but this doesn’t include the file timestamps.

As a result, the only way that seems to be generating consistently reproducible ZIP archives from the same set of files is to force the modification time on files to some well-known time and zip them without any extra attributes. More specifically:

touch -t 201212210314.16 . myfiles.*
zip -X -q myfiles.*

This works consistently (and portably) on OS X and in Linux but of course, there is no guarantee it will stay the same: new zip version might slightly change the algorithm (hopefully not, considering how pervasive ZIP is). But  at least for now, this seems to be the only way to perform reproducible, portable ZIP archive generation.

Posted in Development | Comments Off on Creating reproducible ZIP archives

Мы рождены, чтоб Босха сделать былью




Posted in Uncategorized | Tagged , | Leave a comment

Remote monitoring with Arduino – part 4

This is the final part in the series of articles (parts 1, 2, 3) about building a remote home monitoring system based on Arduino.

Lessons learned

The monitoring task I’ve had was unusual enough to justify building my own system. For more typical tasks, like monitoring the inside/outside temperature, it would be much easier to buy a relatively inexpensive thermometer with a PC interface.
Overall cost of the built system is in the ballpark of some $150-170 – still way below what the commercial systems cost (especially with multiple sensors and flood detection)
When it comes to the power consumption, commercial consumer electronics sensors are way more efficient. A typical temperature sensor can run on a couple of batteries for many months, whereas an Arduino-based device would have to draw AC power.
Wireless communications are painful 🙂 I am yet to look into the message corruption issues described in the previous article.

Future directions

Presentation might need some improvement. Currently Highcharts determines the Y scale dynamically, making relatively small temperature fluctuations look too dramatic. Forcing the scale might make it more meaningful.
An alarm mechanism is needed to alert me on water level change.
It would be great to find a way to read the wireless data emitted by one of the cheap commercial temperature sensors. That way, one could potentially stick a lot of sensors around the place (assuming those come with some notion of a unique ID), have them run on batteries and read them all using a single Arduino device. I will definitely look into this more.
There is a lot of wireless motion open/close sensors on the market operating using different smarthome protocols (X10, ZigBee, etc). Having a way to read and decode their wireless messages would allow extending the monitoring functionality by sending an alarm if one of such sensors is triggered. When it comes to X10, there are two options:
  • Use a USB dongle CM19A that speaks X10. Although it is not 100% certain, this device is apparently capable of receiving messages from X10 wireless sensors.
  • Again, using Arduino with a 433MHz wireless receiver – apparently there is an effort underway to make Arduino parse those messages.
Finally, it would be nice to add more active temperature control for the house but currently it has independent heaters in each room, so it would require somewhat more work in order to switch them to a unified temperature control.
Posted in Smarthome | Tagged , , | Leave a comment

Remote monitoring with Arduino – part 3

Build time! In the previous two articles I’ve discussed the architecture of the remote monitoring solution based on Arduino. Now is the time to build it.


Ideally, it would have been nice to use some kind of sockets to be able to connect and disconnect the sensors but I didn’t bother with this for now. At the end, I settled for the following:

  • DS18B20 is soldered directly onto the shield
  • The thermistor is extended using the two-wire 24-gauge speaker cable (as it needs to run through the crawlspace window outside) which is soldered directly onto the shield
  • DHT22 temperature/humidity sensor is also soldered directly onto the shield, as it doesn’t need to go to far
  • As for the eTape sensor, I ended up using 4-pin JST SM cable. Doing it all over again, I probably would’ve used the speaker wire as well, as only to wires are needed.


shieldArduino protoshields are a neat and convenient way to mount pieces that need to be connected to Arduino. The shields that I used came from CuteDigi (model ARDUINO_PROTO_SHILED_D15 – note the spelling) and while they came assembled, saving me some time to put them together, they turned out to have one nasty problem. Somehow, the numbering of analog pins on the shiled is in reverse to the Arduino board (that is A5 pin on the shield corresponds to A0 on Arduino). Because of this, I’ve gone through a lot of pain trying to figure out why my eTape sensor didn’t work and I literally had to take the entire assembly apart until I found it. Thank you, CuteDigi!

For the enclosure, there aren’t too many Arduino-specific options. I’ve used the fairly standard Arduino Project Enclosure for both devices and it seems to do the job just fine. It’s not perfect: one of the guiding pins that go through the holes in the Arduino board doesn’t have a corresponding hole on the protoshield (I checked 3 different models), so it needed to be cut half-way. Also, the enclosure sports a compartment for the 9V battery despite the fact that the power socket is facing outside. But apart from those issues it works just fine.


Not much to say here apart from showing the code. DS18B20 is handled using the OneWire library. RF communications are done using VirtualWire. eTape and DHT22 are programmed directly (using the tutorial code examples). For eTape, I’m using a simple table-based interpolation within the reasonable range.

The slave device polls its 3 sensors every 15 seconds and sends a message which looks like “M crsp 1234 0 5678”, which means that it is a crawlspace monitor update, that the crawlspace temperature is 12.34 °C (Arduino library sprintf() doesn’t handle floating point formatting), water level of 0 cm, and the outside temperature of 56.78 °C. Sorry, no 5-page XML messages here.

The master device polls its DHT22 sensor and also receives the messages from the slave. Such messages are parsed, checked and combined with device’s own sensor data to form a JSON message which is sent over the serial line. A typical message looks like this:

{ “office_temp” : 24.20 , “office_humidity” : 22.00 }

Crawlspace device code:

// -*- c++ -*-


#undef int
#undef abs
#undef double
#undef float
#undef round

// uncomment to debug
//#define DO_DEBUG

// pins

#define LED_PIN  7              // LED
#define ETAPE_PIN A5            // eTape
#define TEMP_SENSOR_PIN 3       // DS18B20
#define TX_PIN 5                // RF send pin
#define THM_PIN A1              // thermistor pin

#define THM_PULL_RESISTOR   10000

#define DEVICE_ID  "crsp"       // to distinguish between devices

// resistance at 25 degrees C

// temp. for nominal resistance (almost always 25 C)

// how many samples to take and average, more takes longer
// but is more 'smooth'
#define NUMSAMPLES 5

// The beta coefficient of the thermistor (usually 3000-4000)
#define BCOEFFICIENT 3950

// the value of the 'other' resistor
#define SERIESRESISTOR 10000

int samples[NUMSAMPLES];

float get_temperature();
float get_thm_temperature();
float get_water_level();
bool send_data(float temp, float level, float temp2);

//Temperature chip i/o
OneWire ds(TEMP_SENSOR_PIN);  // on digital pin 2

void setup(void)

// setup the transmitter

vw_set_ptt_inverted(true); // Required for RF Link module
vw_setup(2000);                 // Bits per sec

void loop(void)
float temperature = get_temperature();

#ifdef DO_DEBUG
Serial.print("Temp1 = ");

float temp2 = get_thm_temperature();

#ifdef DO_DEBUG
Serial.print("Temp2 = ");

float water_level = get_water_level();

#ifdef DO_DEBUG
Serial.print("Level = ");

send_data(temperature, water_level, temp2);
delay(10000); //just here to slow down the output so it is easier to read

bool send_data(float temp, float level, float temp2)
char msg[128];
snprintf(msg, sizeof(msg), "M %s %d %d %d",
((int) (temp * 100.0)),
((int) level),
((int) (temp2 * 100.0)));

digitalWrite(7, true); // Flash a light to show transmitting
vw_send((uint8_t *)msg, strlen(msg));
vw_wait_tx();              // wait until the whole message is gone

Serial.print("sent: ");

digitalWrite(7, false);
return true;

// returns the temperature from one DS18S20 in Celsius

float get_temperature()
byte data[12];
byte addr[8];

if ( ! {
//no more sensors on chain, reset search
return -1000;

if ( OneWire::crc8( addr, 7) != addr[7]) {
Serial.println("CRC is not valid!");
return -1000;

if ( addr[0] != 0x10 && addr[0] != 0x28) {
Serial.print("Device is not recognized");
return -1000;

ds.write(0x44,1); // start conversion, with parasite power on at the end

byte present = ds.reset();;
ds.write(0xBE); // Read Scratchpad

for (int i = 0; i < 9; i++) { // we need 9 bytes
data[i] =;


byte MSB = data[1];
byte LSB = data[0];

float tempRead = ((MSB << 8) | LSB); //using two's compliment
float TemperatureSum = tempRead / 16;

return TemperatureSum;

// spline-interpolated resistance-to-level table

float r;
float h;
} leveltbl[] = {
{ 747.0000, 1.0000 },
{ 744.9097, 1.1000 },
{ 742.8016, 1.2000 },
{ 740.6732, 1.3000 },
{ 738.5221, 1.4000 },
{ 736.3458, 1.5000 },
{ 734.1419, 1.6000 },
{ 731.9078, 1.7000 },
{ 729.6411, 1.8000 },
{ 727.3393, 1.9000 },
{ 725.0000, 2.0000 },
{ 722.6207, 2.1000 },
{ 720.1989, 2.2000 },
{ 717.7322, 2.3000 },
{ 715.2181, 2.4000 },
{ 712.6542, 2.5000 },
{ 710.0379, 2.6000 },
{ 707.3668, 2.7000 },
{ 704.6384, 2.8000 },
{ 701.8503, 2.9000 },
{ 699.0000, 3.0000 },
{ 696.0865, 3.1000 },
{ 693.1147, 3.2000 },
{ 690.0908, 3.3000 },
{ 687.0213, 3.4000 },
{ 683.9125, 3.5000 },
{ 680.7707, 3.6000 },
{ 677.6022, 3.7000 },
{ 674.4133, 3.8000 },
{ 671.2105, 3.9000 },
{ 668.0000, 4.0000 },
{ 664.7823, 4.1000 },
{ 661.5344, 4.2000 },
{ 658.2274, 4.3000 },
{ 654.8325, 4.4000 },
{ 651.3208, 4.5000 },
{ 647.6635, 4.6000 },
{ 643.8316, 4.7000 },
{ 639.7963, 4.8000 },
{ 635.5287, 4.9000 },
{ 631.0000, 5.0000 },
{ 626.1813, 5.1000 },
{ 621.0437, 5.2000 },
{ 615.5584, 5.3000 },
{ 609.6965, 5.4000 },
{ 603.4292, 5.5000 },
{ 596.7275, 5.6000 },
{ 589.5626, 5.7000 },
{ 581.9056, 5.8000 },
{ 573.7277, 5.9000 },
{ 565.0000, 6.0000 }

int tabsize = sizeof(leveltbl) / sizeof(leveltbl[0]);

interp(float r)
int ii;

for (ii = 0; ii < tabsize; ++ii) {         if (r > leveltbl[ii].r) {
if (ii == 0)
return 0.0;

float h = leveltbl[ii-1].h +
(leveltbl[ii].h - leveltbl[ii-1].h) * (r - leveltbl[ii-1].r) /
(leveltbl[ii].r - leveltbl[ii-1].r);
return h;

return leveltbl[tabsize-1].h;

float get_water_level()
float reading;

reading = analogRead(ETAPE_PIN);

#ifdef DO_DEBUG
Serial.print("eTape analog reading: ");

return interp(reading) * 2.54;

//reading = (1023 / reading) - 1;
//return (SERIESRESISTOR / reading);

float get_thm_temperature()
uint8_t i;
float average = 0;

// take N samples in a row, with a slight delay and average

for (i=0; i< NUMSAMPLES; i++) {
samples[i] = analogRead(THM_PIN);
average += samples[i];

average /= NUMSAMPLES;

#ifdef DO_DEBUG
Serial.print("Thermistor avg reading: ");

// convert the value to resistance
average = 1023 / average - 1;
average = SERIESRESISTOR / average;

#ifdef DO_DEBUG
Serial.print("Thermistor resistance: ");

float steinhart;
steinhart = average / THERMISTORNOMINAL;     // (R/Ro)
steinhart = log(steinhart);                  // ln(R/Ro)
steinhart /= BCOEFFICIENT;                   // 1/B * ln(R/Ro)
steinhart += 1.0 / (TEMPERATURENOMINAL + 273.15); // + (1/To)
steinhart = 1.0 / steinhart;                 // Invert
steinhart -= 273.15;                         // convert to C

return steinhart;

Receiver code

// -*- c++ -*-

#undef int
#undef abs
#undef double
#undef float
#undef round
#include "DHT.h"

#define DHT_PIN 4

// Uncomment whatever type you're using!
#define DHTTYPE DHT22   // DHT 22  (AM2302)


int tokenize(char* s, const char* sep, char* tokens[], int maxtoks);
int parse_crawlspace_data(const char* s, float* temp, float* level,
                          float* temp2);

// uncomment to show trace
//#define SHOW_TRACE

int count = 0;
int first_time = 1;


    // Initialise the IO and ISR
    vw_set_ptt_inverted(true);    // Required for RX Link Module
    vw_setup(2000);                   // Bits per sec
    vw_set_rx_pin(7);           // We will be receiving on pin 23 (Mega) ie the RX pin from the module connects to this pin.
    vw_rx_start();                      // Start the receiver

    char remote_msg[VW_MAX_MESSAGE_LEN * sizeof(uint8_t) + 1];
    uint8_t buf[VW_MAX_MESSAGE_LEN];
    uint8_t buflen = VW_MAX_MESSAGE_LEN;
    int got_remote_data = 0, got_local_data = 0, rc;
    float temp, level, temp2;

    //Serial.println("Waiting for incoming");

    if (first_time) {
        for (int ii = 0; ii < 150; ii++) { #ifdef SHOW_TRACE             Serial.println("Waiting for message"); #endif                      if (vw_get_message(buf, &buflen)) { #ifdef SHOW_TRACE                 Serial.println("Got 1st good message"); #endif                 break;             }             delay(100);         }         first_time = 0;     }          //rc = vw_wait_rx_max(16000);      #ifdef SHOW_TRACE     //Serial.print("Finished waiting: ");     //Serial.println(rc); #endif     if (vw_get_message(buf, &buflen)) {         strncpy(remote_msg, (const char*) buf, sizeof(remote_msg));         int slen = buflen * sizeof(uint8_t);         if (slen >= sizeof(remote_msg))
            slen = sizeof(remote_msg) - 1;

        remote_msg[slen] = '\0';

        if (parse_crawlspace_data(remote_msg, &temp, &level, &temp2) == 0)
            got_remote_data = 1;
        else {
            Serial.print("DEBUG: ignoring message: ");

        Serial.print("DEBUG: got message: ");
    else {

    Serial.println("No message received");


    // Reading temperature or humidity takes about 250 milliseconds!
    // Sensor readings may also be up to 2 seconds 'old' (its a very slow sensor)
    float local_humidity = dht.readHumidity();
    float local_temp = dht.readTemperature();

    // check if returns are valid, if they are NaN (not a number) then something went wrong!
    if (isnan(local_temp) || isnan(local_humidity)) {
        Serial.println("DEBUG: failed to read from DHT");
    else {
        got_local_data = 1;

        Serial.print("DEBUG: Humidity: ");
        Serial.print(" %\t");
        Serial.print("Temperature: ");
        Serial.println(" *C");

    if (got_local_data && got_remote_data) {
        Serial.print("{ \"office_temp\" : ");
        Serial.print(" , \"office_humidity\" : ");
        Serial.print(" , \"cr_temp\" : ");
        Serial.print(" , \"cr_water\" : ");
        Serial.print(" , \"out_temp\" : ");
        Serial.println(" }");
    else if (got_local_data && count > 30) {
        Serial.print("{ \"office_temp\" : ");
        Serial.print(" , \"office_humidity\" : ");
        Serial.println(" }");
        count = 0;


parse_crawlspace_data(const char* s, float* temp, float* level, float* temp2)
    char scopy[64];
    strncpy(scopy, s, sizeof(scopy));
    scopy[sizeof(scopy) - 1] = '\0';

    char* tokens[16];
    int ntoks = tokenize(scopy, " \t", tokens,
                         sizeof(tokens) / sizeof(tokens[0]));

    if (ntoks != 5 || strcmp(tokens[0], "M") || strcmp(tokens[1], "crsp"))
        return -1;

    int val = atoi(tokens[2]);
    *temp = ((float) val) / 100.00;

    val = atoi(tokens[3]);

    if (val >= 0)
        *level = val;
        *level = -100;

    val = atoi(tokens[4]);
    *temp2 = ((float) val) / 100.00;

    return 0;

tokenize(char* s, const char* sep, char* tokens[], int maxtoks)
    char *ptr = s;
    char* rest = NULL;

    for (int ii = 0; ii < maxtoks; ii++) {
        tokens[ii] = strtok_r(ptr, sep, &rest);

        if (tokens[ii] == NULL)
            return ii;

        ptr = rest;

    return maxtoks;


Being new to the game, I was unpleasantly surprised that the devices that worked pretty well when sitting next to each other, stopped communicating when moved to different rooms. Turned out the communication module was pretty weak. Fortunately, running a simple antennae gave a necessary boost in reception. I used 1.5 ft lengths of 20-gauge wire for both devices to get it to work reasonably well when separated by walls or floors.

More communication woes

Apart from the signal strength issues, I ran into more insidious communication problems. These happened on a number of occasions and went away eventually – I still don’t know what might have been causing them. In those cases, the receiver device would not receive messages sent by the slave. When that happened, the receiver would receive a lot of messages from the sender but most of them would be corrupted in one way or another, so the VirtualWave would reject them as they fail the CRC check. I still don’t know what might be causing them, so at the end, I simply disabled the CRC check in VirtualWave relying on parsing of the data on the master device which would reject obviously malformed messages. This is far from perfect but better than having no communication at all.



One last problem discovered only after mounting the slave device was that the water level readings reported by the eTape ended up indicating 2cm water level 🙂 . As the eTape sensor is extremely well… sensitive to its shape and internal forces, I believe this is due to the way it is being suspended by relatively stiff wire (i.e. it is not perfectly vertical). Later I intend to attach it using a piece of dual-side sticky tape but for now, I’ll pretend that 2 cm of water in the crawlspace is normal. After all, I am more interested in a change of level rather than in the level itself.

Collecting and publishing the data



The master Arduino device is connected by USB to a small form-factor “server” (NetTop nT-535) I’m running 24×7. I’ve been using this PC for a while but if I started from scratch, I would definitely consider using Raspberry Pi. Since I need to process the data and to encrypt them prior to sending over the Net (yes, I am that paranoid), making Arduino post data on the Net directly (i.e. via the Ethernet shield) is impractical.

The “server” runs Ubuntu LTS (server edition). Sensor monitoring functionality consists of a Python script that perform all the tasks and an Upstart config file for that script, registering it as a service managed by Upstart. The script also throttles the data flow only posting the new readings every 15 minutes.

Both files are presented here (with the actual data posting functionality omitted)

#!/usr/bin/env python

# get sensor data from Arduino and upload to the webservice

import datetime
import re
import os
import os.path
import logging
import sys
import time
import hashlib
import random
from optparse import OptionParser
import simplejson as json
from Crypto.Cipher import AES
import pycurl
import StringIO
import serial

def redirect_output(file):
    f =, 1, 0644)

    if os.dup(f) != 1:
        sys.stderr.write("ERROR: failed to dup STDOUT\n")


    if os.dup(f) != 2:
        sys.stderr.write("ERROR: failed to dup STDERR\n")

usage_string = """Usage:  %prog [options] <image_file>

%prog takes an image, encrypts and uploads it to the web server.


prog = os.path.basename(sys.argv[0])
program_version = None          # default: use RCS ID

if not(program_version):
    program_version = "$Revision: 1.1$"
    program_version = re.sub("\$.evision:\s*(\S*)\s*\$$", "\g",

version_string = "%%prog  %s" % program_version

# parse command-line options

parser = OptionParser(usage = usage_string,
                      version = version_string)
parser.add_option("-U", "--url",
                  help = "upload URL",
                  metavar = "URL", dest = "url", default = upload_url)
parser.add_option("-l", "--log",
                  help = "redirect output to a log file",
                  metavar = "FILE", dest = "log_file")
parser.add_option("-r", help = "Send unencrypted data",
                  action = "store_true", dest = "unencrypted", default = False)
parser.add_option("-v", "--verbose", help = "verbose operation",
                  action = "store_true", dest = "verbose_mode")

(options, args) = parser.parse_args()

# redirect output

if options.log_file :

if options.verbose_mode :
    logging.basicConfig(format = "%(asctime)s  " + prog +
                        ": %(levelname)s: %(message)s",
                        level = logging.DEBUG)
else :
    logging.basicConfig(format = "%(asctime)s  " + prog +
                        ": %(levelname)s: %(message)s",
                        level = logging.INFO)

# detect the serial device

for ii in (0, 1, 2, 3, 4, 5):
    dev = "/dev/ttyACM%d" % ii

    if os.path.exists(dev):
        logging.debug("using device %s" % dev)
        ser = serial.Serial(dev, 9600)

last_sent_time = - datetime.timedelta(hours=1)

while (True) :
        status = ser.readline()

        logging.error("exception when reading serial line (disconnected?)")

    logging.debug("received status: %s" % status)

    if re.match(r'^\s*$', status) :

    obj = None

        obj = json.loads("""{ "readings" : %s }""" % status)

        logging.error("malformed status string: %s" % status)

    obj['time'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
    full_data = 'cr_temp' in obj
    data = json.dumps(obj)

    if not options.unencrypted:
        data = encrypt_data(data, ...)

    delta = - last_sent_time

    if (full_data and delta.seconds > 900) or \
            (not full_data and delta.seconds > 1000):
        logging.debug("sending %s data after %d seconds" % \
                          ((full_data and "full" or "short"), delta.seconds))
        last_sent_time =
    else :
        logging.debug("not sending  - too early (%d)" % delta.seconds)



Eventually, the readings end up in a MySQL database, from where they are fetched by the presentation side, which consists of a relatively simple PHP script. The actual charting is done by the excellent Highcharts package.

The end result

With everything working almost as expected, I am now able to see the all the data on a single page:


In the next (and last) part, I will talk about lessons learned and future directions and enhancements that could be applied to this system.

Posted in Smarthome | Tagged , , | Leave a comment

Remote monitoring with Arduino – part 2

In part 1 I have looked at the options available for the task of remote climate monitoring and settled on Arduino-based solution. In this we will discuss the design of the system.

General architecture

The high-level architecture is typical for Arduino: there is an Arduino-based board that collects the sensor data and passes it over USB to the PC that processes this data and uploads it to the database. A more parsimonious solution would be to build an Arduino board with an Ethernet or Wi-Fi shield and to connect it directly to the network, but I already had a tiny Atom-based PC (NetBox nT-535) running 24×7 and doing other tasks. Plus, Arduino library is fairly limited, so doing any more or less sophisticated data manipulation right on Arduino is difficult.

Since I needed to monitor 3 different locations (outside, crawlspace, and indoors), I could either run wires or work out some kind of wireless solution to communicate the data from different places. Wired solution is easy electronically but requires drilling holes and running wires around the place which is not fun. A good solution would not need that, so wireless was indeed the answer. In this case, I could do away with running 2 Arduinos: one sitting in the crawlspace and collecting the crawlspace temperature, outdoor temperature, checking the water level on the ground, and sending all this data to the “master” Arduino board that collected the indoor temperature and sent all collected data to the PC via USB.

Wireless communication

Fortunately, Arduino has a number of options when it comes to wireless communications: from WiFi to Bluetooth, to ZigBee. However most of these options are a big overkill. What is really needed is something similar to what those cheap indoor/outdoor thermometers are using, which is low-power RF communication. Fortunately, Arduino has plenty of options in this department as well. An inexpensive pair of a 315MHz receiver and transmitter can provide uni-directional communication between the “slave” and “master” units. The one that I settled on was a pair of modules from Sparkfun. They are small, cheap, and relatively easy to use.



Since one of the objectives was to have fun and try different things out, I didn’t want to use the same model of the temperature sensor in three different places, instead opting for the following ones:

  • Environmentally protected DS18B20 digital temperature sensor for crawlspace temperature monitoring.
  • Analog 10K thermistor for outdoor monitoring (one of the reasons to use this model is that it runs only 2 wires so it is easier to extend its reach)
  • Digital DHT22 temperature/humidity sensor for indoor monitoring (as a bonus I get the indoor humidity to observe).

Flood detection however is a completely different story. Flood sensors on the market are few and apart and they are not geared toward Arduino. Whereas the simplest flood sensor can be built very easily by running two wires that get shorted when covered by water, this is too volatile and crude (let alone the fact that it requires some salt in order to ensure conductivity). There are also some fairly complex mechanical options based on measuring the level of flotation of the moving part.

etapeThe one I finally settled on was an eTape liquid level sensor. In short, it is a variable resistor that reacts to the water pressure, which is exactly what I was looking for. On the downside, it is extremely sensitive to its placement, shape and inner tensions and it is also not cheap (about $40).

Final architecture

In its final iteration, the system consists of two Arduino boards. A “slave” board sits in the crawlspace. It uses AC power via a 5V AC adapter and it runs 3 sensors:

  • Inside (crawlspace) temperature (DS18B20)
  • Outside temperature (Thermistor)
  • Water level (eTape)

The slave board periodically queries all sensors and sends all data via RF wireless to the “master” board, sitting inside the house. The master board has only one temperature/humidity sensor (DHT22). It combines all data and sends it via serial communication (over the USB) to the PC that performs further processing.

In the next part we will look into the actual implementation and issues I had to solve while building it.

Posted in Smarthome | Tagged , , | Leave a comment

Remote monitoring with Arduino – part 1

Being a paranoid person, I always wanted to monitor our summer house remotely. With all the advancement of technology, one would think such a task is easy these days but surprisingly it is not. Off-the-shelf systems look like they are still stuck in the 80s and no one has bothered to build an open and modular system for such a purpose.

In the coming several posts I’ll describe the system I’ve built for my own purposes. Hopefully, this would be helpful for the next guy trying to do the same.

The problem

Defining the problem is half of the task. While there are many things one could monitor, for starters, I decided to stick to the following monitors:

  • Indoor temperature
  • Outdoor temperature
  • Crawlspace temperature
  • Crawlspace flood monitoring

The options

There is a wide gamut of monitoring options available. Each one has its own pros and cons. What follows is a very basic benefit analysis for each of the most obvious ones. Granted, such analysis is subjective and depends on your goals and circumstances.

Off-the-shelf alarm system

There are tons of those on the market wit varying degree of sophistication. The biggest attraction is that such systems only require installation and relatively simple configuration. They also come with versatile sensors, including motion/open sensors, video cameras, and climate sensors. However they have several disadvantages:

  • Only relatively sophisticated models come with flood detection.
  • Such systems are not open and can only be extended in very limited ways.
  • Only very advanced systems come with a PC/internet interface.
  • Most of such systems require a phone line and a subscription to a monitoring service.
  • They have relatively high cost.

Consumer-grade weather stations with PC/network interface

Ambient Weather WS-2080These are significantly less expensive (on an average) than the security systems. On an upside, they are a lot of fun, allowing to monitor various weather conditions, capture them for your own analysis (in Linux as well – see the excellent WView software) and uploading to the community sites like Weather Underground. As for the disadvantages, they are as follows:

  • Most of consumer-grade systems are not open. They come at most with two temperature sensors and the models that allow more are significantly more expensive.
  • None of them come with flood detection.
  • Quality is an issue. Most of affordable models have a lot of negative feedback related to quality and reliability.

Hacked indoor/outdoor thermometer

ThermometerCheap indoor/outdoor thermometers are a dime a dozen these days. If there is a way to hack into the indoor unit to capture the data, that could be a terrific solution (for the temperature monitoring at least). Sensors are very cheap, they run off a couple of AA batteries for 5-6 months and they are small enough to be placed anywhere. The problem is, these devices are not really open to tinkering and my non-existent electronics skills are clearly insufficient for reverse engineering such a device.

Another  way to make use of such sensors is to build an Arduino-based receiver that is capable of reading off-the-shelf temperature sensors. This is doable, at least in theory and there are a couple of efforts underway to build such receivers. This is something I will definitely look into later as it would simplify the system dramatically.

DIY Arduino-based system

ArduinoOn the upside, doing it is a lot of fun and allows a lot of flexibility: one can build wired or wireless monitoring, it is possible to monitor a lot of different sensors and conditions. Also, Arduino is relatively inexpensive.

On the downside, small as Arduino is, it is still a relative overkill for temperature monitoring. Where an off-the-shelf temperature sensor could run on a couple of AA batteries for half a year, an Arduino based solution would require an AC adapter. And of course it is only suitable if one is not afraid to do some handiwork, to solder components, and to troubleshoot things when they don’t work right away.

As the title suggest, this is the option I ultimately settled on 🙂 . In the part 2, I will discuss the design of the system.

Posted in Smarthome | Tagged , , | Leave a comment