Using Jinja templates with AJAX calls

In a Flask framework, one typically renders JINJA templates directly from Python router:

@app.route("/users")
def show_users():
    user_list = get_user_list()
    return render_template("user_list.html",
           user_list)

This is fine but what if one uses AJAX to add interactivity? Data returned via AJAX is returned to the client side (Javascript), whereas Python code (and thus template rendering) runs server-side. Surely, one can render HTML on the client side directly (or utilize one of the JS template frameworks) but now rendering logic is scattered across different layers, which makes it a nightmare to maintain.

One way to simplify this is to make the API called by AJAX return an already rendered (via Jinja) HTML.

Suppose, you have a REST API to fetch the list of users:

@app.route("/api/v1/get_users", methods=["GET"])
def get_users():
    users = { 0: "root", 42: "bwk" }
    return(jsonify(users))

One could define an additional code path through this API to return the result pre-rendered as HTML:

@app.route("/api/v1/get_users", methods=["GET"])
@app.route("/api/v1/get_users/<do_render>",
           methods=["GET"])
def get_users(do_render=None):
    users = { 0: "root", 42: "bwk" }

    if do_render:
        return jsonify({ "rendered" : render_template("user_list_fragment.html",
           users) })
    else:
        return(jsonify(users))

..and then render it directly in Javascript:

$.getJSON("http://.../api/v1/get_users",
    true, 
    function(data) {
        $('#user-list').html(data["rendered"]);
    } 
});

VS Code Dark+ theme for Emacs

Being a life-long proponent of dark mode for everything (starting from VT 220 and Wyse terminals decades ago), I do tend to experiment with various dark themes every now and then. Until recently, I’ve been sticking to the awesome Dracula theme (which at this point exists for pretty much every program under the Sun), but I’ve discovered that the Dark+ theme for Visual Studio Code is even more appealing to me – it is reasonably constrained, trying not to be too flashy for its own good – overall, a pleasure to use.

Probably because of its relatively young age, it hasn’t been ported to many other programs, most notably, to Emacs. So, this sounded like a win-win situation: an opportunity to play with colors, to create a new color theme, and to learn something new in the process – what’s not to like?!

So here we go – a brand new vscdark theme for Emacs! There is a pull request for MELPA repo – once it is approved, the new theme will be readily available via package-install (for now, you have to download it from Github).

Managing e-statements for fun and profit

Unless you’ve been living under a rock for the last 5 years, you should have noticed the imminent extinction of paper statements from most of the consumer services. Whether we like it or not, e-bills are the way of the future.

One problem with e-bills is that they still need to be stored – preferably, in a way that makes them easy to find. One also cannot rely on the service owners to archive old bills, as they rarely keep them for longer than 1 year.

A typical process managing such bills involves multiple steps:

  • Download the new bill from the website of the service provider.
  • Rename the PDF file from an exciting automatic (e.g 8B96766CBC0A49E6A45C2B039623C56C.pdf) to a boring one (e.g. TRowePrice-2017-Q3.pdf)
  • Store it so that one could find it when needed (ideally, not in a single folder holding thousands of such files).

About 3 years ago, I’ve written a relatively simple tool that attempts to automate the latter two of those 3 steps (automatically downloading is notoriously difficult, given different website organization for each provider and the fact that they love to tweak the website layout, frequently breaking the scraping procedure).

The tool is called classify_bills and is available on GitHub. When run on a bunch on PDF files, it tries to figure out what account each file belongs to and what time period it is for. It then renames it appropriately and moves it into an appropriate destination.

The tool is configuration-driven, it basically runs off a bunch of regular expressions (hat tip to jwz), and configuring a new account is strictly speaking, not pleasant. Still, once configured, it eliminates the chore of laboriously doing those actions manually, bill by bill, month by month, making the process faster and pleasanter.

Using Raspberry Pi to monitor and water plants

Introduction

I’ve been growing hot peppers on my balcony and one of the things I constantly fail to keep on schedule is watering them. As it is with many chores, this is something simple yet not exciting enough to avoid being actively procrastinated on.

As such, I only saw several solutions to that problem:

  • Become better at avoiding procrastination (yeah, ri-i-i-ight!)
  • Outsource the activity to some hapless family member.
  • Get rid of the peppers.
  • Throw technology (and time, and money) at the problem.

Given all those options, the choice was rather simple. As a result, I built a computer-driven watering station that takes care of my peppers for me. At this point, my involvement is only needed to periodically add water to the water tank (about once every 2-3 weeks). Apart from that, the system is fully self-driven.

In its current incarnation, the system handles 4 pots (number of pots is limited by current hardware – in theory it should be possible to add more relays and moisture sensors). It periodically measures moisture level of the soil in each of the pots. When it decides the soil is “dry enough” it waters the plant. It also makes sure watering has the desired effect (i.e. preventing conditions like tank running empty or watering tube being misplaced). Finally, it posts all kinds of stats on a dashboard. (not yet, actually).

Continue reading “Using Raspberry Pi to monitor and water plants”

Rotating between different fonts in Emacs

There are two types of people: those who set on one font/color theme for the rest of their life and those like me, who constantly tweak their setup. Not that it helps productivity, but hey – RMS created .emacs for a reason!

Since typing in font names over and over again is a pain, I’ve created a neat function that can be called interactively to rotate between various programming-friendly fonts installed in the system. The list of fonts is partly mine, partly borrowed from Programming Fonts  (which is awesome on its own). This allows a very quick way to see how one’s code looks with different fonts (and make a final decision for the next 3 weeks or so).

Continue reading “Rotating between different fonts in Emacs”

Creating reproducible ZIP archives

An Artist A DayI am currently working on an app that loads data in relatively small incremental payloads from the server side for offline use. Payload files are in ZIP format and the way the app knows whether it has the correct file (which might not be true – the file might be missing, partially downloaded or simply changed by developer) is by checking it’s checksum against the local file. If there is a checksum mismatch, the file is (re)downloaded.

During development, I regenerate the payload files many times and when the time comes, I release the “golden” versions into production. Obviously, I insist on reproducible builds and this includes the bundle files – if the bits of the data that went into those files haven’t changed, there should be absolutely no reason why the generated bundle should differ in a single bit. After all if the file has a new signature, the app will download it again (even if the data is the same). I could devise a more elaborate way via some kind of versioning but it would be more complex, more error-prone, and wouldn’t add much benefit to the user. Plus, it’s nice to know that your data is guaranteed to be the same.

Image credit: New Line Cinema.
Image credit: New Line Cinema.

Turns out, things are not so easy when it comes to ZIP files. Both OS X and Ubuntu offer command-line zip program which by default sticks gobs of metadata into the ZIP archive, including timestamps. Thus, no two zips are alike (well, maybe they will be if run within the same second, but I didn’t bother to check).

Unfortunately there is no way to tell zip to only include most critical metadata (file names and sizes) and ignore everything else. It does offer -X (aka -no-extra) option to strip some attributes but this doesn’t include the file timestamps.

As a result, the only way that seems to be generating consistently reproducible ZIP archives from the same set of files is to force the modification time on files to some well-known time and zip them without any extra attributes. More specifically:

touch -t 201212210314.16 . myfiles.*
zip -X -q myarchive.zip myfiles.*

This works consistently (and portably) on OS X and in Linux but of course, there is no guarantee it will stay the same: new zip version might slightly change the algorithm (hopefully not, considering how pervasive ZIP is). But  at least for now, this seems to be the only way to perform reproducible, portable ZIP archive generation.

Remote monitoring with Arduino – part 4

This is the final part in the series of articles (parts 1, 2, 3) about building a remote home monitoring system based on Arduino.  

Lessons learned

The monitoring task I’ve had was unusual enough to justify building my own system. For more typical tasks, like monitoring the inside/outside temperature, it would be much easier to buy a relatively inexpensive thermometer with a PC interface.   Overall cost of the built system is in the ballpark of some $150-170 – still way below what the commercial systems cost (especially with multiple sensors and flood detection)   When it comes to the power consumption, commercial consumer electronics sensors are way more efficient. A typical temperature sensor can run on a couple of batteries for many months, whereas an Arduino-based device would have to draw AC power.   Wireless communications are painful 🙂 I am yet to look into the message corruption issues described in the previous article.  

Future directions

Presentation might need some improvement. Currently Highcharts determines the Y scale dynamically, making relatively small temperature fluctuations look too dramatic. Forcing the scale might make it more meaningful.   An alarm mechanism is needed to alert me on water level change.   It would be great to find a way to read the wireless data emitted by one of the cheap commercial temperature sensors. That way, one could potentially stick a lot of sensors around the place (assuming those come with some notion of a unique ID), have them run on batteries and read them all using a single Arduino device. I will definitely look into this more.   There is a lot of wireless motion open/close sensors on the market operating using different smarthome protocols (X10, ZigBee, etc). Having a way to read and decode their wireless messages would allow extending the monitoring functionality by sending an alarm if one of such sensors is triggered. When it comes to X10, there are two options:  

  • Use a USB dongle CM19A that speaks X10. Although it is not 100% certain, this device is apparently capable of receiving messages from X10 wireless sensors.
  • Again, using Arduino with a 433MHz wireless receiver – apparently there is an effort underway to make Arduino parse those messages.

Finally, it would be nice to add more active temperature control for the house but currently it has independent heaters in each room, so it would require somewhat more work in order to switch them to a unified temperature control.

Remote monitoring with Arduino – part 3

Wiring

Build time! In the previous two articles I’ve discussed the architecture of the remote monitoring solution based on Arduino. Now is the time to build it.

Ideally, it would have been nice to use some kind of sockets to be able to connect and disconnect the sensors but I didn’t bother with this for now. At the end, I settled for the following:

  • DS18B20 is soldered directly onto the shield
  • The thermistor is extended using the two-wire 24-gauge speaker cable (as it needs to run through the crawlspace window outside) which is soldered directly onto the shield
  • DHT22 temperature/humidity sensor is also soldered directly onto the shield, as it doesn’t need to go to far
  • As for the eTape sensor, I ended up using 4-pin JST SM cable. Doing it all over again, I probably would’ve used the speaker wire as well, as only to wires are needed.
Continue reading “Remote monitoring with Arduino – part 3”