Managing e-statements for fun and profit

Unless you’ve been living under a rock for the last 5 years, you should have noticed the imminent extinction of paper statements from most of the consumer services. Whether we like it or not, e-bills are the way of the future.

One problem with e-bills is that they still need to be stored – preferably, in a way that makes them easy to find. One also cannot rely on the service owners to archive old bills, as they rarely keep them for longer than 1 year.

A typical process managing such bills involves multiple steps:

  • Download the new bill from the website of the service provider.
  • Rename the PDF file from an exciting automatic (e.g 8B96766CBC0A49E6A45C2B039623C56C.pdf) to a boring one (e.g. TRowePrice-2017-Q3.pdf)
  • Store it so that one could find it when needed (ideally, not in a single folder holding thousands of such files).

About 3 years ago, I’ve written a relatively simple tool that attempts to automate the latter two of those 3 steps (automatically downloading is notoriously difficult, given different website organization for each provider and the fact that they love to tweak the website layout, frequently breaking the scraping procedure).

The tool is called classify_bills and is available on GitHub. When run on a bunch on PDF files, it tries to figure out what account each file belongs to and what time period it is for. It then renames it appropriately and moves it into an appropriate destination.

The tool is configuration-driven, it basically runs off a bunch of regular expressions (hat tip to jwz), and configuring a new account is strictly speaking, not pleasant. Still, once configured, it eliminates the chore of laboriously doing those actions manually, bill by bill, month by month, making the process faster and pleasanter.

Using Raspberry Pi to monitor and water plants


I’ve been growing hot peppers on my balcony and one of the things I constantly fail to keep on schedule is watering them. As it is with many chores, this is something simple yet not exciting enough to avoid being actively procrastinated on.

As such, I only saw several solutions to that problem:

  • Become better at avoiding procrastination (yeah, ri-i-i-ight!)
  • Outsource the activity to some hapless family member.
  • Get rid of the peppers.
  • Throw technology (and time, and money) at the problem.

Given all those options, the choice was rather simple. As a result, I built a computer-driven watering station that takes care of my peppers for me. At this point, my involvement is only needed to periodically add water to the water tank (about once every 2-3 weeks). Apart from that, the system is fully self-driven.

In its current incarnation, the system handles 4 pots (number of pots is limited by current hardware – in theory it should be possible to add more relays and moisture sensors). It periodically measures moisture level of the soil in each of the pots. When it decides the soil is “dry enough” it waters the plant. It also makes sure watering has the desired effect (i.e. preventing conditions like tank running empty or watering tube being misplaced). Finally, it posts all kinds of stats on a dashboard. (not yet, actually).

Continue reading “Using Raspberry Pi to monitor and water plants”

Rotating between different fonts in Emacs

There are two types of people: those who set on one font/color theme for the rest of their life and those like me, who constantly tweak their setup. Not that it helps productivity, but hey – RMS created .emacs for a reason!

Since typing in font names over and over again is a pain, I’ve created a neat function that can be called interactively to rotate between various programming-friendly fonts installed in the system. The list of fonts is partly mine, partly borrowed from Programming Fonts  (which is awesome on its own). This allows a very quick way to see how one’s code looks with different fonts (and make a final decision for the next 3 weeks or so).

Continue reading “Rotating between different fonts in Emacs”

Creating reproducible ZIP archives

An Artist A DayI am currently working on an app that loads data in relatively small incremental payloads from the server side for offline use. Payload files are in ZIP format and the way the app knows whether it has the correct file (which might not be true – the file might be missing, partially downloaded or simply changed by developer) is by checking it’s checksum against the local file. If there is a checksum mismatch, the file is (re)downloaded.

During development, I regenerate the payload files many times and when the time comes, I release the “golden” versions into production. Obviously, I insist on reproducible builds and this includes the bundle files – if the bits of the data that went into those files haven’t changed, there should be absolutely no reason why the generated bundle should differ in a single bit. After all if the file has a new signature, the app will download it again (even if the data is the same). I could devise a more elaborate way via some kind of versioning but it would be more complex, more error-prone, and wouldn’t add much benefit to the user. Plus, it’s nice to know that your data is guaranteed to be the same.

Image credit: New Line Cinema.
Image credit: New Line Cinema.

Turns out, things are not so easy when it comes to ZIP files. Both OS X and Ubuntu offer command-line zip program which by default sticks gobs of metadata into the ZIP archive, including timestamps. Thus, no two zips are alike (well, maybe they will be if run within the same second, but I didn’t bother to check).

Unfortunately there is no way to tell zip to only include most critical metadata (file names and sizes) and ignore everything else. It does offer -X (aka -no-extra) option to strip some attributes but this doesn’t include the file timestamps.

As a result, the only way that seems to be generating consistently reproducible ZIP archives from the same set of files is to force the modification time on files to some well-known time and zip them without any extra attributes. More specifically:

touch -t 201212210314.16 . myfiles.*
zip -X -q myfiles.*

This works consistently (and portably) on OS X and in Linux but of course, there is no guarantee it will stay the same: new zip version might slightly change the algorithm (hopefully not, considering how pervasive ZIP is). But  at least for now, this seems to be the only way to perform reproducible, portable ZIP archive generation.

Remote monitoring with Arduino – part 4

This is the final part in the series of articles (parts 1, 2, 3) about building a remote home monitoring system based on Arduino.  

Lessons learned

The monitoring task I’ve had was unusual enough to justify building my own system. For more typical tasks, like monitoring the inside/outside temperature, it would be much easier to buy a relatively inexpensive thermometer with a PC interface.   Overall cost of the built system is in the ballpark of some $150-170 – still way below what the commercial systems cost (especially with multiple sensors and flood detection)   When it comes to the power consumption, commercial consumer electronics sensors are way more efficient. A typical temperature sensor can run on a couple of batteries for many months, whereas an Arduino-based device would have to draw AC power.   Wireless communications are painful 🙂 I am yet to look into the message corruption issues described in the previous article.  

Future directions

Presentation might need some improvement. Currently Highcharts determines the Y scale dynamically, making relatively small temperature fluctuations look too dramatic. Forcing the scale might make it more meaningful.   An alarm mechanism is needed to alert me on water level change.   It would be great to find a way to read the wireless data emitted by one of the cheap commercial temperature sensors. That way, one could potentially stick a lot of sensors around the place (assuming those come with some notion of a unique ID), have them run on batteries and read them all using a single Arduino device. I will definitely look into this more.   There is a lot of wireless motion open/close sensors on the market operating using different smarthome protocols (X10, ZigBee, etc). Having a way to read and decode their wireless messages would allow extending the monitoring functionality by sending an alarm if one of such sensors is triggered. When it comes to X10, there are two options:  

  • Use a USB dongle CM19A that speaks X10. Although it is not 100% certain, this device is apparently capable of receiving messages from X10 wireless sensors.
  • Again, using Arduino with a 433MHz wireless receiver – apparently there is an effort underway to make Arduino parse those messages.

Finally, it would be nice to add more active temperature control for the house but currently it has independent heaters in each room, so it would require somewhat more work in order to switch them to a unified temperature control.

Remote monitoring with Arduino – part 3


Build time! In the previous two articles I’ve discussed the architecture of the remote monitoring solution based on Arduino. Now is the time to build it.

Ideally, it would have been nice to use some kind of sockets to be able to connect and disconnect the sensors but I didn’t bother with this for now. At the end, I settled for the following:

  • DS18B20 is soldered directly onto the shield
  • The thermistor is extended using the two-wire 24-gauge speaker cable (as it needs to run through the crawlspace window outside) which is soldered directly onto the shield
  • DHT22 temperature/humidity sensor is also soldered directly onto the shield, as it doesn’t need to go to far
  • As for the eTape sensor, I ended up using 4-pin JST SM cable. Doing it all over again, I probably would’ve used the speaker wire as well, as only to wires are needed.
Continue reading “Remote monitoring with Arduino – part 3”

Remote monitoring with Arduino – part 2

In part 1 I have looked at the options available for the task of remote climate monitoring and settled on Arduino-based solution. In this we will discuss the design of the system.

General architecture

The high-level architecture is typical for Arduino: there is an Arduino-based board that collects the sensor data and passes it over USB to the PC that processes this data and uploads it to the database. A more parsimonious solution would be to build an Arduino board with an Ethernet or Wi-Fi shield and to connect it directly to the network, but I already had a tiny Atom-based PC (NetBox nT-535) running 24×7 and doing other tasks. Plus, Arduino library is fairly limited, so doing any more or less sophisticated data manipulation right on Arduino is difficult.

Since I needed to monitor 3 different locations (outside, crawlspace, and indoors), I could either run wires or work out some kind of wireless solution to communicate the data from different places. Wired solution is easy electronically but requires drilling holes and running wires around the place which is not fun. A good solution would not need that, so wireless was indeed the answer. In this case, I could do away with running 2 Arduinos: one sitting in the crawlspace and collecting the crawlspace temperature, outdoor temperature, checking the water level on the ground, and sending all this data to the “master” Arduino board that collected the indoor temperature and sent all collected data to the PC via USB.

Continue reading “Remote monitoring with Arduino – part 2”

Remote monitoring with Arduino – part 1

Being a paranoid person, I always wanted to monitor our summer house remotely. With all the advancement of technology, one would think such a task is easy these days but surprisingly it is not. Off-the-shelf systems look like they are still stuck in the 80s and no one has bothered to build an open and modular system for such a purpose.

In the coming several posts I’ll describe the system I’ve built for my own purposes. Hopefully, this would be helpful for the next guy trying to do the same.

Continue reading “Remote monitoring with Arduino – part 1”