SensorMote Project 3/31/13
After watching the Distributed Environmental Data: On the Ground at the Data Sensing Lab video I was inspired to take on a larger project. I also thought it would be a nice distraction from my other projects. Up to this point my focus was audio, but sensors are also an interest. Using sensors in tandem with sound is an important part of what I am studying. So a sensor-specific project was definitely something I wanted to tackle.
This has proven to be the most complex project so far, with some firsts… It’s the first time I used Python, the XBee wireless network platform, and (despite social media overload) I even joined Google+ specifically to gain access to some more info on the project. Overall it’s been worth the effort.
For picts and some videos:
Continue to read a more detailed account.
Before I go further, I need to thank Brian Jepson (who was involved with an early revision of the SensorMote project) and Alasdair Allan, one of the original designers, who both turned out to be accessible and very patient with me as I got deeper down this rabbit hole.
I’m going to touch on a few stumbling blocks here that will hopefully save others some trouble. Of course, I started with the GitHub site for the project (https://github.com/aallan/SensorMote) and downloaded everything. The materials list indicates Arduino Leonardo and XBee Series 2, but keep in mind that is what they used at the conference because they were working with a larger mesh network. As we will see, Uno and Xbee Series 1 are really all that is required to get this project off the ground for the home project builder.
Alasdair notes “The sketches should work just fine with an Leonardo, although they’ll need modification due to the Serial and Serial1 issue to talk to the XBee, at the Leonardo uses a software serial port to talk to the computer rather than the hardware one the Uno uses, see http://arduino.cc/en/Reference/Serial, and the comments there about the Leonardo.”
I ordered all of my sensors from Adafruit, and a Wireless Shield from Maker Shed. After looking over the project a few times to conceptualize what I was doing, I decided to spring for a digital copy of Alasdair’s book, Distributed Network Data – From hardware to data to visualization (cowritten with Kipp Bradford).
The book serves as a (mostly) complete guide to the project they built for the conference. One discovery – and this was also a first for me – is that a digital book on an iPad is 100% better to use than a paper copy when it comes to these projects. Not only was I able to more easily read through everything and hyperlink out to some references, but when it came time to build, it was invaluable to have the quick/easy scroll/zoom ability. I won’t build another complex project without this resource.
The other extremely helpful shortcut is that you can copy/paste the code examples. I know, hard code fanatics will shake their fingers at me for this, and yes, it may be the lazy way out if you want to be a coder. But I do not. I want to understand what is happening conceptually, and get right to seeing it work. My interest is in the finished product, and what can be done from there.
Fear not – I got burned by this approach at least twice :-)
I’ll break this project down into the following main components:
- Sensors
- Output to File
- Wireless Connectivity
- Visualization (making the data appear in dynamic graphs)
SENSORS
My sensors arrived quickly from Adafruit, and I already had an Arduino UNO and breadboard to get started. The project starts with just getting the humidity/temp sensor connected. No problems with the Arduino code. As soon as I added the PIR, that’s where the trouble started. In both the PIR and MIC sketches, I got these compile errors:
SensorDHTPIRMIC:42: error: stray ‘\’ in program
SensorDHTPIRMIC.ino: In function ‘void loop()’:
SensorDHTPIRMIC:42: error: ‘u22121’ was not declared in this scope
After a couple hours, a break, and some web research, I finally found the problem. When I copied the code directly from the PDF book, one of the lines was rendered as:
motionState = −1;
The dash character pulled from the PDF is a long hyphen. Arduino wants to see a short hyphen:
motionState = -1;
To the eye the difference is very subtle, and easily missed, but to the machine they are completely different. With this problem out of the way, I was able to blow through the rest of the sketches, all the way up to the point of adding LEDs and making the output machine readable.
OUTPUT TO FILE
To make the serial data machine readable, this project requires a an additional programming piece that takes us outside of the Arduino world. The authors use Python, and this was new territory for me. Conceptually I understood what it would do, but just getting a workable environment was a hassle. And while Alasdair’s book is very good with just about everything else, it assumes a great deal about the reader’s skill level. A little more guidance – or specific links to specific resources – would have been appreciated.
All of my project build up to this point was done from my electronics workbench where I have an old HP laptop running Win XP. It’s just a beater PC, but it works well enough for most basic tasks. Alasdair’s instructions are specific to the Mac world, which is fine since most of my research and audio activity takes place on a MacBook. But since I wanted to do everything in one place, I opted to try the Python piece in the PC realm. I do not recommend this, but here it goes…
PC Notes
1. Installed Python 3.3
2. Installed pySerial – At first I used pyserial-py3k-2.5.win32.exe because that is what came up on the site as most recent. As we’ll see, that could explain some of the issues I had, and others have had the same issue.
In the Distributed Network Data book, the command to find a port is:
python -m serial.tools.list_ports
On the pySerial site (http://pyserial.sourceforge.net/) they note that this command “will print a list of available ports. It is also possible to add a regexp as first argument and the list will only include entries that matched. The enumeration may not work on all operating systems. It may be incomplete, list unavailable ports or may lack detailed descriptions of the ports.”
OK, not very encouraging, but I proceeded anyway:
>>> -m serial.tools.list_ports
SyntaxError: invalid syntax
(“serial” was highlighted as the problem area)
>>> python -m serial.tools.list_ports
SyntaxError: invalid syntax
>>> serial.tools.list_ports
Traceback (most recent call last):
File “<pyshell#4>”, line 1, in <module>
serial.tools.list_ports
AttributeError: ‘module’ object has no attribute ‘tools’
I already knew I was on COM4 because the Arduino IDE tells you, and you can also look in Control Panel to find out where a device is connected – all GUI and much easier to deal with – so I took a wild guess at the PC syntax:
>>> import serial
>>> ser = serial.Serial(‘COM4’)
>>> x = ser.read(30)
All good up to there…
>>> print x
SyntaxError: invalid syntax
After some web research:
>>> print (x)
b’21.10, 28.40, 3, 0, 0\r\n21.10, ‘
Then don’t forget to close:
ser.close()
So this was encouraging. Entering the commands line by line, I got something that resembled the correct output. I am sure the b, \r\n stuff will be a problem later though (see update below).
Moving on to see if this can be run as a small program all together, I copy/pasted all of the Python code into a text file, saved with PY extension. And that’s where copy/paste burned me again. Python is sensitive to indents, but this was not 100% clear at first, and I wasted a lot of time spinning my wheels. This is my main problem with hard coding in these types of language. There is no feedback to the user, or when there is, it’s cryptic and relies on some prior secret knowledge.
Even with correct indents, Line 12 caused a problem:
print a
It returns a syntax error, but no help on what to do to fix it. It’s similar to the error I received when entering the commands line by line, so I tried:
print (a)
That too failed. After some web research on the differences in syntax between Python versions, and some guessing, I ended up with:
print ((a))
This gave me output as follows:
30/03/2013 09:07:22, b’20.90, 28.60, 3, 0, 0\r\n’,
30/03/2013 09:07:23, b”,
30/03/2013 09:07:24, b”,
30/03/2013 09:07:25, b’20.90, 28.60, 5, 0, 0\r\n’,
30/03/2013 09:07:26, b”,
These notes for the PC workflow represent several hours or work.
Now let’s move on to the Mac issues.
- Installed Python 3.3
- Installed pySerial 2.6 – This was the most difficult part, and requires the Mac equivalent of the DOS prompt – Terminal. Go to http://georgesattler.id.au/archives/679 and save yourself some time. I love my Mac as much as the next guy, but for all the user-friendliness that Mac’s are known for, I found the process of installing a programming environment hideously difficult. Thanks again George.
- Create text file (TextPad or Pages, but needs to be plain text with PY extension).
- Copy/paste code from PDF book, make corrections to indents as needed.
- Open Python IDLE, open PY file, run it. See if it crashes.
Python on Mac continually crashed for me, but I eventually got to a point where I could run the module. I also didn’t get the error on Line 12 as I did with the PC, and I had a better understanding of syntax issues to look for before going too far. This gave me the best output:
28/03/2013 16:56:47, 21.130, 30.50, 5, 0, 0,
28/03/2013 17:01:19, .40, 6, 0, 0,
28/03/2013 17:05:27, ,
28/03/2013 17:11:26, 20.20, 1, 0, 0,
28/03/2013 17:12:20, ,
28/03/2013 17:12:39, 21.0, 24, 0, 0,
28/03/2013 19:07:26, ,
28/03/2013 19:07:26, 21.40, 31.50, 61, 0, 0,
28/03/2013 19:07:27, ,
28/03/2013 19:07:28, ,
28/03/2013 19:07:29, 21.50, 31.50, 0, 0, 0,
I had hoped that by the time I got through this part of the project, I’d have the other parts I ordered. The Maker Shed transaction was a big disappointment. My order sat for several days, and it wasn’t until I contacted them about the delay that I learned the Wireless Shield was on backorder for 5 weeks. They canceled the order, and I wasn’t charged, but it delayed the wireless portion of this project by about a week. So I used Amazon – it’s the same price, and you get free shipping.
So by the looks of it, I’m half-way through this project. Time to go play some guitar and take a break!
Oh, and another helpful resource is the code tester at http://ideone.com/
UPDATE on PC Setup
Fixing PC Python pySerial install:
Installing pySerial on Windows 7 (works for WinXP as well)
The \r\n issue…
This turned out to be Python inconsistency that others have seen. I still do not have the programming skill to correct this, so I will likely work on Mac for the rest of this project.
Saturday April 6 – Progress, a Python Hang problem, and a bad Xbee :-(
I returned to the SensorMote project after a week of other distractions, and waiting for parts to come in. I left off with Chapter 6, “Chapter 6. Simplifying the Design”, and a large part of the chapter is dedicated to moving the breadboard design to an Arduino Protoshield. Having worked with such shields before, most of this chapter was review.
The new part for me was the last sketch that outlines how to program additional power pins for connecting additional components. This sketch is confusing because in addition to the new lines about power, there is a new section:
// Check that we have a reading from the DHT sensor
if (isnan(t) || isnan(h)) {
Serial.println(“Error: Failed to read from DHT”);
This section introduces feedback on an error. In the serial monitor it look like:
Read failError: Failed to read from DHT
1,1,73
22.30,21.20,0,-1,4
22.80,21.20,1,1,30
22.70,21.20,1,0,9
22.70,21.20,1,0,4
22.80,21.20,0,-1,2
22.90,21.20,0,0,3
22.90,21.20,0,0,1
This error caused me to do additional troubleshooting. It is missing from previous sketches in the project. Each time, the error would occur only once – at start up of the serial monitor. It would not repeat. After considerable time, I couldn’t find anything wrong, per se. No explanation in the book. It seems in this Chapter 6 sketch, there is an expected delay in the DHT sensor output, and the built in error is just there to… who knows. On to the Xbee wireless portion of the project.
WIRELESS
Per the instructions in Chapter 7, “Chapter 7. Building Point-to-Point XBee Networks,” all of the Xbee configuration is done on a Mac using a combination of the Mac native terminal program (here I’ve labeled this “Mac Terminal”), native Python command line environment (which is accessed via Mac Terminal), and the CoolTerm terminal. It somewhat amazes me that three “terminals” are needed to do this project, and it has been… terminally challenging.
Further clarification:
Mac Terminal – Use for checking serial ports on the Mac, installing pySerial, and for other troubleshooting of the Python install.
Python – Use for creating the file that will store values coming from the SensorMote (making the machine-readable output).
CoolTerm – Use for configuring the Xbee devices and replacing the USB cable to the SensorMote.
The Xbee Adapters from Adafruit came as a kit, but within an hour I had them both built. The step regarding careful – very light – soldering of the Xbee headers is critical. I just barely avoided permanently soldering one of my Xbees into an adapter.
With the Xbees built, I was able to clearly see the serial connection in Mac Terminal
ls /dev/tty.*
This returns:
/dev/tty.usbserial-FTAK1ILS
CoolTerm setup was quirky, and it took some tweaking because the book is not 100% accurate.
Problem #1 – One would assume that the Xbee and the connector cable used for programming would be flush on one side. However this is not the case. There is no guidance in the book on this. The image of the Xbee and Adafruit adapter is very poor, and doesn’t help. The right-most pin of the Xbee is not supposed to be connected. I only discovered this via XBee Basics – Lesson 1 – General Information and Initial Setup – http://www.youtube.com/watch?v=odekkumB3WQ
Problem #2 – Also, in CoolTerm, Key Emulation needs to be CR, which I only learned from Xbee series 1 setup using CoolTerm in OSX
http://www.youtube.com/watch?v=o2zdwXfY56I
Check it out on my Flickr page. From there however, it was all downhill. Python would not save the data to a file, and eventually I was getting only a time stamp, and no sensor data. Then I noted Python would hang:
This prompted a wild goose chase with my Python & pySerial install, only to find one of my Xbees was bad. It completely failed (no serial communication). Only a blinking green light. Many hours of troubleshooting later (including attempts to “un-brick” the Xbee), I finally ruled out everything else and contacted Adafruit for a replacement. Thankfully they are pretty good about stuff like this.
Unbrick links that I tried (but none helped):
Tutorial: How to unbrick, restore your brocken Xbee / Zigbee devices
The Unofficial XBee FAQ (version 1.3)
Most of the tutorials on the web assume that you have an “interface board” with a reset button. But the Adafruit Xbee Adapter has no such reset button, and these instructions do not work, even hot wiring the Reset and Ground connections – as some have claimed – did not work for me.
5/10/13 Update
After about three weeks, I came back to this (we are in the process of moving, and other priorities took over). Adafruit sent the replacement Xbee quickly, and to my surprise when I finally sat down to work on this, progress was swift. Before long I retracted my steps with Coolterm, got the new Xbee programmed, and got into the Processing side of things for the graphing.
My Python GUI (IDLE) installation still has trouble recognizing the serial library. Luckily this isn’t an issue to truly proceed. This is where working ahead somewhat can help. Eventually, the tasks that Python was performing get replaced by the Processing environment, and instead of working with a stored file (which is one of the intermediate examples), the goal is to work with live data. At least that is what I was most interested in doing.
When I was still working with a saved file, the first sketch in Processing works, but only if the extra lines with commas are removed from the test.csv.
10/05/2013 17:44:49, 44.20,27.00,0,0,8
,
10/05/2013 17:44:49, 43.90,27.00,0,0,5
,
10/05/2013 17:44:49, 43.70,27.00,0,0,5
,
10/05/2013 17:44:49, 43.6043.70,27.20,0,0,7
,
10/05/2013 17:44:50, ,
Proceeding with the Processing exercises, I noticed that Exercise 2 – show temperature on graph – is missing a right paren:
PFont f = createFont(“Arial”,16,true);
Something doesn’t look right though. Graph is blank, which I first interpretted as a program error. All of my data was time stamped as of the same second, so I tried incrementing these entries, but that didn’t work. So I just moved on…
The examples eventually drop using the CSV file, and migrate to reading directly from the serial connection. For the temp exercise, I found the temp was actually reading at over 40 degrees C, so nothing would display on the graph because the scale only went to 40. I took an ice pack out of the freezer and put it near my sensor. Here’s the result:
The next exercise – mapping humidity and temp together – took a little tweaking. As I understood the code, I was able to modify the scale.
From here I did some manual editing and troubleshooting and before long had temp and humidity plotting simultaneously. See graphic. It appears that one errant value is plotted for a time, along the bottom of the graph. Still need to figure out where that is coming from. But temp (around 50 C) and humidity (around 24%) are clearly plotted correctly according to the scales I’ve set up.
This is a major step, and I am stopping to take a break while I am doing well. Check out a couple videos from this segment on Flickr:
Dec 2013 Update – After many months away from Arduino due to a move, I came back to this project and incredibly it worked despite being jostled around quite a bit. I messed with some of the values in the processing program to pull in the other sensor data, and at one point got four plots of different sizes/colors.
The real limitation is that the scales for each value (temp, sound, humidity, movement) do not really have a common thread, so displaying them all at once on one graph is kind of silly without some serious scaling and labeling, which is currently beyond me.
The next step for this project is to get the circuit off the breadboard, but I am taking a step back to get my workbench in order before I do that.
