Remote monitoring with Arduino (part 3)
February 2013 (5121 Words, 29 Minutes)
Wiring
Build time! In the previous two articles I’ve discussed the architecture of the remote monitoring solution based on Arduino. Now is the time to build it.
Ideally, it would have been nice to use some kind of sockets to be able to connect and disconnect the sensors but I didn’t bother with this for now. At the end, I settled for the following:
- DS18B20 is soldered directly onto the shield
- The thermistor is extended using the two-wire 24-gauge speaker cable (as it needs to run through the crawlspace window outside) which is soldered directly onto the shield
- DHT22 temperature/humidity sensor is also soldered directly onto the shield, as it doesn’t need to go to far
- As for the eTape sensor, I ended up using 4-pin JST SM cable. Doing it all over again, I probably would’ve used the speaker wire as well, as only to wires are needed.
Assembly
Arduino protoshields are a neat and convenient way to mount pieces that need to be connected to Arduino. The shields that I used came from CuteDigi (model ARDUINO_PROTO_SHILED_D15 – note the spelling) and while they came assembled, saving me some time to put them together, they turned out to have one nasty problem. Somehow, the numbering of analog pins on the shiled is in reverse to the Arduino board (that is A5 pin on the shield corresponds to A0 on Arduino). Because of this, I’ve gone through a lot of pain trying to figure out why my eTape sensor didn’t work and I literally had to take the entire assembly apart until I found it. Thank you, CuteDigi!
For the enclosure, there aren’t too many Arduino-specific options. I’ve used the fairly standard Arduino Project Enclosure for both devices and it seems to do the job just fine. It’s not perfect: one of the guiding pins that go through the holes in the Arduino board doesn’t have a corresponding hole on the protoshield (I checked 3 different models), so it needed to be cut half-way. Also, the enclosure sports a compartment for the 9V battery despite the fact that the power socket is facing outside. But apart from those issues it works just fine.
Programming
Not much to say here apart from showing the code. DS18B20 is handled using the OneWire library. RF communications are done using VirtualWire. eTape and DHT22 are programmed directly (using the tutorial code examples). For eTape, I’m using a simple table-based interpolation within the reasonable range.
The slave device polls its 3 sensors every 15 seconds and sends a
message which looks like “M crsp 1234 0 5678”, which means that it
is a crawlspace monitor update, that the crawlspace temperature is
12.34°C (Arduino library sprintf() doesn’t handle floating point
formatting), water level of 0 cm, and the outside temperature of
56.78°C. Sorry, no 5-page XML messages here.
The master device polls its DHT22 sensor and also receives the messages from the slave. Such messages are parsed, checked and combined with device’s own sensor data to form a JSON message which is sent over the serial line. A typical message looks like this:
{ “office_temp” : 24.20 , “office_humidity” : 22.00 }
Crawlspace device code:
// -*- c++ -*-
#undef int
#undef abs
#undef double
#undef float
#undef round
// uncomment to debug
//#define DO_DEBUG
// pins
#define LED_PIN 7 // LED
#define ETAPE_PIN A5 // eTape
#define TEMP_SENSOR_PIN 3 // DS18B20
#define TX_PIN 5 // RF send pin
#define THM_PIN A1 // thermistor pin
#define THM_PULL_RESISTOR 10000
#define DEVICE_ID "crsp" // to distinguish between devices
// resistance at 25 degrees C
#define THERMISTORNOMINAL 10000
// temp. for nominal resistance (almost always 25 C)
#define TEMPERATURENOMINAL 25
// how many samples to take and average, more takes longer
// but is more 'smooth'
#define NUMSAMPLES 5
// The beta coefficient of the thermistor (usually 3000-4000)
#define BCOEFFICIENT 3950
// the value of the 'other' resistor
#define SERIESRESISTOR 10000
int samples[NUMSAMPLES];
float get_temperature();
float get_thm_temperature();
float get_water_level();
bool send_data(float temp, float level, float temp2);
// Temperature chip i/o
OneWire ds(TEMP_SENSOR_PIN); // on digital pin 2
void
setup(void)
{
Serial.begin(9600);
// setup the transmitter
Serial.println("setup");
vw_set_ptt_inverted(true); // Required for RF Link module
vw_setup(2000); // Bits per sec
vw_set_tx_pin(TX_PIN);
}
void
loop(void)
{
float temperature = get_temperature();
#ifdef DO_DEBUG
Serial.print("Temp1 = ");
Serial.println(temperature);
#endif
float temp2 = get_thm_temperature();
#ifdef DO_DEBUG
Serial.print("Temp2 = ");
Serial.println(temp2);
#endif
float water_level = get_water_level();
#ifdef DO_DEBUG
Serial.print("Level = ");
Serial.println(water_level);
#endif
send_data(temperature, water_level, temp2);
delay(10000); // just here to slow down the output so it is easier to read
}
bool
send_data(float temp, float level, float temp2)
{
char msg[128];
snprintf(msg,
sizeof(msg),
"M %s %d %d %d",
DEVICE_ID,
((int)(temp * 100.0)),
((int)level),
((int)(temp2 * 100.0)));
digitalWrite(7, true); // Flash a light to show transmitting
vw_send((uint8_t*)msg, strlen(msg));
vw_wait_tx(); // wait until the whole message is gone
Serial.print("sent: ");
Serial.println(msg);
delay(200);
digitalWrite(7, false);
delay(200);
return true;
}
// returns the temperature from one DS18S20 in Celsius
float
get_temperature()
{
byte data[12];
byte addr[8];
if (!ds.search(addr)) {
// no more sensors on chain, reset search
ds.reset_search();
return -1000;
}
if (OneWire::crc8(addr, 7) != addr[7]) {
Serial.println("CRC is not valid!");
return -1000;
}
if (addr[0] != 0x10 && addr[0] != 0x28) {
Serial.print("Device is not recognized");
return -1000;
}
ds.reset();
ds.select(addr);
ds.write(0x44, 1); // start conversion, with parasite power on at the end
byte present = ds.reset();
ds.select(addr);
ds.write(0xBE); // Read Scratchpad
for (int i = 0; i < 9; i++) { // we need 9 bytes
data[i] = ds.read();
}
ds.reset_search();
byte MSB = data[1];
byte LSB = data[0];
float tempRead = ((MSB << 8) | LSB); // using two's compliment
float TemperatureSum = tempRead / 16;
return TemperatureSum;
}
// spline-interpolated resistance-to-level table
struct
{
float r;
float h;
} leveltbl[] = {
{ 747.0000, 1.0000 }, { 744.9097, 1.1000 }, { 742.8016, 1.2000 },
{ 740.6732, 1.3000 }, { 738.5221, 1.4000 }, { 736.3458, 1.5000 },
{ 734.1419, 1.6000 }, { 731.9078, 1.7000 }, { 729.6411, 1.8000 },
{ 727.3393, 1.9000 }, { 725.0000, 2.0000 }, { 722.6207, 2.1000 },
{ 720.1989, 2.2000 }, { 717.7322, 2.3000 }, { 715.2181, 2.4000 },
{ 712.6542, 2.5000 }, { 710.0379, 2.6000 }, { 707.3668, 2.7000 },
{ 704.6384, 2.8000 }, { 701.8503, 2.9000 }, { 699.0000, 3.0000 },
{ 696.0865, 3.1000 }, { 693.1147, 3.2000 }, { 690.0908, 3.3000 },
{ 687.0213, 3.4000 }, { 683.9125, 3.5000 }, { 680.7707, 3.6000 },
{ 677.6022, 3.7000 }, { 674.4133, 3.8000 }, { 671.2105, 3.9000 },
{ 668.0000, 4.0000 }, { 664.7823, 4.1000 }, { 661.5344, 4.2000 },
{ 658.2274, 4.3000 }, { 654.8325, 4.4000 }, { 651.3208, 4.5000 },
{ 647.6635, 4.6000 }, { 643.8316, 4.7000 }, { 639.7963, 4.8000 },
{ 635.5287, 4.9000 }, { 631.0000, 5.0000 }, { 626.1813, 5.1000 },
{ 621.0437, 5.2000 }, { 615.5584, 5.3000 }, { 609.6965, 5.4000 },
{ 603.4292, 5.5000 }, { 596.7275, 5.6000 }, { 589.5626, 5.7000 },
{ 581.9056, 5.8000 }, { 573.7277, 5.9000 }, { 565.0000, 6.0000 }
};
int tabsize = sizeof(leveltbl) / sizeof(leveltbl[0]);
float
interp(float r)
{
int ii;
for (ii = 0; ii < tabsize; ++ii) {
if (r > leveltbl[ii].r) {
if (ii == 0)
return 0.0;
float h =
leveltbl[ii - 1].h + (leveltbl[ii].h - leveltbl[ii - 1].h) *
(r - leveltbl[ii - 1].r) /
(leveltbl[ii].r - leveltbl[ii - 1].r);
return h;
}
}
return leveltbl[tabsize - 1].h;
}
float
get_water_level()
{
float reading;
reading = analogRead(ETAPE_PIN);
#ifdef DO_DEBUG
Serial.print("eTape analog reading: ");
Serial.println(reading);
#endif
return interp(reading) * 2.54;
// reading = (1023 / reading) - 1;
// return (SERIESRESISTOR / reading);
}
float
get_thm_temperature()
{
uint8_t i;
float average = 0;
// take N samples in a row, with a slight delay and average
for (i = 0; i < NUMSAMPLES; i++) {
samples[i] = analogRead(THM_PIN);
average += samples[i];
delay(10);
}
average /= NUMSAMPLES;
#ifdef DO_DEBUG
Serial.print("Thermistor avg reading: ");
Serial.println(average);
#endif
// convert the value to resistance
average = 1023 / average - 1;
average = SERIESRESISTOR / average;
#ifdef DO_DEBUG
Serial.print("Thermistor resistance: ");
Serial.println(average);
#endif
float steinhart;
steinhart = average / THERMISTORNOMINAL; // (R/Ro)
steinhart = log(steinhart); // ln(R/Ro)
steinhart /= BCOEFFICIENT; // 1/B * ln(R/Ro)
steinhart += 1.0 / (TEMPERATURENOMINAL + 273.15); // + (1/To)
steinhart = 1.0 / steinhart; // Invert
steinhart -= 273.15; // convert to C
return steinhart;
}
Receiver code:
// -*- c++ -*-
#undef int
#undef abs
#undef double
#undef float
#undef round
#include "DHT.h"
#define DHT_PIN 4
// Uncomment whatever type you're using!
#define DHTTYPE DHT22 // DHT 22 (AM2302)
DHT dht(DHT_PIN, DHTTYPE);
int
tokenize(char* s, const char* sep, char* tokens[], int maxtoks);
int
parse_crawlspace_data(const char* s, float* temp, float* level, float* temp2);
// uncomment to show trace
//#define SHOW_TRACE
int count = 0;
int first_time = 1;
void
setup()
{
Serial.begin(9600);
// Initialise the IO and ISR
vw_set_ptt_inverted(true); // Required for RX Link Module
vw_setup(2000); // Bits per sec
vw_set_rx_pin(7); // We will be receiving on pin 23 (Mega) ie the RX pin
// from the module connects to this pin.
vw_rx_start(); // Start the receiver
dht.begin();
}
void
loop()
{
char remote_msg[VW_MAX_MESSAGE_LEN * sizeof(uint8_t) + 1];
uint8_t buf[VW_MAX_MESSAGE_LEN];
uint8_t buflen = VW_MAX_MESSAGE_LEN;
int got_remote_data = 0, got_local_data = 0, rc;
float temp, level, temp2;
#ifdef SHOW_TRACE
// Serial.println("Waiting for incoming");
#endif
if (first_time) {
for (int ii = 0; ii < 150; ii++) {
#ifdef SHOW_TRACE Serial.println("Waiting for message");
#endif if (vw_get_message(buf, &buflen))
{
#ifdef SHOW_TRACE Serial.println("Got 1st good message");
#endif break;
}
delay(100);
}
first_time = 0;
} // rc = vw_wait_rx_max(16000); #ifdef SHOW_TRACE
// //Serial.print("Finished waiting: "); //Serial.println(rc); #endif
// if (vw_get_message(buf, &buflen)) { strncpy(remote_msg, (const
// char*) buf, sizeof(remote_msg)); int slen = buflen *
// sizeof(uint8_t); if (slen >= sizeof(remote_msg))
slen = sizeof(remote_msg) - 1;
remote_msg[slen] = '\0';
if (parse_crawlspace_data(remote_msg, &temp, &level, &temp2) == 0)
got_remote_data = 1;
else {
Serial.print("DEBUG: ignoring message: ");
Serial.println(remote_msg);
}
#ifdef SHOW_TRACE
Serial.print("DEBUG: got message: ");
Serial.print(remote_msg);
Serial.println("");
#endif
}
else
{
#ifdef SHOW_TRACE
Serial.println("No message received");
#endif
}
// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds 'old' (its a very slow sensor)
float local_humidity = dht.readHumidity();
float local_temp = dht.readTemperature();
// check if returns are valid, if they are NaN (not a number) then something
// went wrong!
if (isnan(local_temp) || isnan(local_humidity)) {
Serial.println("DEBUG: failed to read from DHT");
} else {
got_local_data = 1;
#ifdef SHOW_TRACE
Serial.print("DEBUG: Humidity: ");
Serial.print(local_humidity);
Serial.print(" %\t");
Serial.print("Temperature: ");
Serial.print(local_temp);
Serial.println(" *C");
#endif
}
if (got_local_data && got_remote_data) {
Serial.print("{ \"office_temp\" : ");
Serial.print(local_temp);
Serial.print(" , \"office_humidity\" : ");
Serial.print(local_humidity);
Serial.print(" , \"cr_temp\" : ");
Serial.print(temp);
Serial.print(" , \"cr_water\" : ");
Serial.print(level);
Serial.print(" , \"out_temp\" : ");
Serial.print(temp2);
Serial.println(" }");
} else if (got_local_data && count > 30) {
Serial.print("{ \"office_temp\" : ");
Serial.print(local_temp);
Serial.print(" , \"office_humidity\" : ");
Serial.print(local_humidity);
Serial.println(" }");
count = 0;
}
delay(500);
count++;
}
int
parse_crawlspace_data(const char* s, float* temp, float* level, float* temp2)
{
char scopy[64];
strncpy(scopy, s, sizeof(scopy));
scopy[sizeof(scopy) - 1] = '\0';
char* tokens[16];
int ntoks =
tokenize(scopy, " \t", tokens, sizeof(tokens) / sizeof(tokens[0]));
if (ntoks != 5 || strcmp(tokens[0], "M") || strcmp(tokens[1], "crsp"))
return -1;
int val = atoi(tokens[2]);
*temp = ((float)val) / 100.00;
val = atoi(tokens[3]);
if (val >= 0)
*level = val;
else
*level = -100;
val = atoi(tokens[4]);
*temp2 = ((float)val) / 100.00;
return 0;
}
int
tokenize(char* s, const char* sep, char* tokens[], int maxtoks)
{
char* ptr = s;
char* rest = NULL;
for (int ii = 0; ii < maxtoks; ii++) {
tokens[ii] = strtok_r(ptr, sep, &rest);
if (tokens[ii] == NULL)
return ii;
ptr = rest;
}
return maxtoks;
}
Reception
Being new to the game, I was unpleasantly surprised that the devices that worked pretty well when sitting next to each other, stopped communicating when moved to different rooms. Turned out the communication module was pretty weak. Fortunately, running a simple antennae gave a necessary boost in reception. I used 1.5 ft lengths of 20-gauge wire for both devices to get it to work reasonably well when separated by walls or floors.
More communication woes
Apart from the signal strength issues, I ran into more insidious communication problems. These happened on a number of occasions and went away eventually – I still don’t know what might have been causing them. In those cases, the receiver device would not receive messages sent by the slave. When that happened, the receiver would receive a lot of messages from the sender but most of them would be corrupted in one way or another, so the VirtualWave would reject them as they fail the CRC check. I still don’t know what might be causing them, so at the end, I simply disabled the CRC check in VirtualWave relying on parsing of the data on the master device which would reject obviously malformed messages. This is far from perfect but better than having no communication at all.
One last problem discovered only after mounting the slave device was that the water level readings reported by the eTape ended up indicating 2cm water level 🙂 . As the eTape sensor is extremely well… sensitive to its shape and internal forces, I believe this is due to the way it is being suspended by relatively stiff wire (i.e. it is not perfectly vertical). Later I intend to attach it using a piece of dual-side sticky tape but for now, I’ll pretend that 2 cm of water in the crawlspace is normal. After all, I am more interested in a change of level rather than in the level itself.
Collecting and publishing the data
The master Arduino device is connected by USB to a small form-factor “server” (NetTop nT-535) I’m running 24×7. I’ve been using this PC for a while but if I started from scratch, I would definitely consider using Raspberry Pi. Since I need to process the data and to encrypt them prior to sending over the Net (yes, I am that paranoid), making Arduino post data on the Net directly (i.e. via the Ethernet shield) is impractical.
The “server” runs Ubuntu LTS (server edition). Sensor monitoring functionality consists of a Python script that perform all the tasks and an Upstart config file for that script, registering it as a service managed by Upstart. The script also throttles the data flow only posting the new readings every 15 minutes.
Both files are presented here (with the actual data posting functionality omitted)
#!/usr/bin/env python
# get sensor data from Arduino and upload to the webservice
import datetime
import re
import os
import os.path
import logging
import sys
import time
import hashlib
import random
from optparse import OptionParser
import simplejson as json
from Crypto.Cipher import AES
import pycurl
import StringIO
import serial
def redirect_output(file):
f = os.open(options.log_file, 1, 0644)
os.close(1)
if os.dup(f) != 1:
sys.stderr.write("ERROR: failed to dup STDOUT\n")
os.close(2)
if os.dup(f) != 2:
sys.stderr.write("ERROR: failed to dup STDERR\n")
usage_string = """Usage: %prog [options] <image_file>
%prog takes an image, encrypts and uploads it to the web server.
"""
prog = os.path.basename(sys.argv[0])
program_version = None # default: use RCS ID
if not(program_version):
program_version = "$Revision: 1.1$"
program_version = re.sub("\$.evision:\s*(\S*)\s*\$$", "\g",
program_version)
version_string = "%%prog %s" % program_version
# parse command-line options
parser = OptionParser(usage=usage_string,
version=version_string)
parser.add_option("-U", "--url",
help="upload URL",
metavar="URL", dest="url", default=upload_url)
parser.add_option("-l", "--log",
help="redirect output to a log file",
metavar="FILE", dest="log_file")
parser.add_option("-r", help="Send unencrypted data",
action="store_true", dest="unencrypted", default=False)
parser.add_option("-v", "--verbose", help="verbose operation",
action="store_true", dest="verbose_mode")
(options, args) = parser.parse_args()
# redirect output
if options.log_file:
redirect_output(options.log_file)
if options.verbose_mode:
logging.basicConfig(format="%(asctime)s " + prog +
": %(levelname)s: %(message)s",
level=logging.DEBUG)
else:
logging.basicConfig(format="%(asctime)s " + prog +
": %(levelname)s: %(message)s",
level=logging.INFO)
# detect the serial device
for ii in (0, 1, 2, 3, 4, 5):
dev = "/dev/ttyACM%d" % ii
if os.path.exists(dev):
logging.debug("using device %s" % dev)
ser = serial.Serial(dev, 9600)
break
last_sent_time = datetime.datetime.now() - datetime.timedelta(hours=1)
while (True):
try:
status = ser.readline()
except:
logging.error("exception when reading serial line (disconnected?)")
sys.exit(1)
logging.debug("received status: %s" % status)
if re.match(r'^\s*$', status):
continue
obj = None
try:
obj = json.loads("""{ "readings" : %s }""" % status)
except:
logging.error("malformed status string: %s" % status)
continue
obj['time'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
full_data = 'cr_temp' in obj
data = json.dumps(obj)
if not options.unencrypted:
data = encrypt_data(data, ...)
delta = datetime.datetime.now() - last_sent_time
if (full_data and delta.seconds > 900) or \
(not full_data and delta.seconds > 1000):
logging.debug("sending %s data after %d seconds" %
((full_data and "full" or "short"), delta.seconds))
send_data(data.encode("base64"))
last_sent_time = datetime.datetime.now()
else:
logging.debug("not sending - too early (%d)" % delta.seconds)
sys.exit()
Presentation
Eventually, the readings end up in a MySQL database, from where they are fetched by the presentation side, which consists of a relatively simple PHP script. The actual charting is done by the excellent Highcharts package.
The end result
With everything working almost as expected, I am now able to see the all the data on a single page:
In the next (and last) part, I will talk about lessons learned and future directions and enhancements that could be applied to this system.