LoRa is great when there’s a network available, but for in-person water quality readings I thought it’d be convenient to add Bluetooth.
I had a spare BLE Nano v2 sitting around, so decided to wire that in. It acts as a relay, broadcasting everything that the Water Quality sensor outputs over serial, and pushing it out via Bluetooth.
Because this is the same setup as my air quality sensor project, I could clone the Air Quality iOS app to become a Water Quality iOS app. The app saves all of the data to my iPhone/iPad’s Core Database, and there’s an export to CSV function too.
It’s pretty snug on the mini breadboard, and I cheated a bit by adding in the 3.3v BLE board on the wrong side of the power isolator to avoid having to drop down the 5v TX pin from the Arduino side to 3.3v, but it all still fits in the waterproof enclosure and functions very nicely.
The Bluetooth BLE board is the red one on the right.
Next step is to update my Rails site to be able to include water quality data, add some API authentication, and host it. Then all of the data the iOS apps are getting will be posted to a database and graphed nicely, both on a linear graph, and on a map.
Sending: mac join otaa
Join not accepted: denied
Check your coverage, keys and backend status.
I’d tried five different locations across two states, so was getting pretty paranoid that it was either something I’d done in my Arduino water quality sensor reading code (I was using Software Serial).
So I asked Leo from TTN Adelaide if it might be possible for him to send me a known working Uno so that I could try it here using stock software. His setup was identical to the one I had been trying, but at least we could rule out hardware failure, and it’d tell me a lot more about trusting my code.
As soon as it arrived I plugged it into my USB power brick so that my laptop couldn’t do anything weird to it (super paranoid), but it didn’t connect successfully. So after a couple of minutes I plugged it into my laptop so I could read the serial monitor output. Same output as mine.
So at this point I could at least feel a little better about my code and hardware, but it was still a bit of a mystery why all of the locations I’d tried didn’t work.
I wandered around the Melbourne CBD with it switched on, monitoring TTN console on my phone for uploads… still nothing.
In a last ditch effort to find some signal, I got myself onto a 38th floor rooftop, switched it on, and almost instantly the join confirmation and payloads started streaming in.
So it was just blackspots all over the Melbourne CBD, East St Kilda, Southbank, Adelaide CBD and Port Willunga.
Test the payload
Next thing to do was to test the payload to see if I was sending a readable byte array.
-- SENSOR READING
EC:78.23
TDS:42
SAL:0.00
GRAV:1.000
-- BYTE ARRAY TO SEND
37 38 2E 32 33 2C 34 32 2C 30 2E 30 30 2C 31 2E 30 30 30 00
Sending: mac tx uncnf 1 37382E32332C34322C302E30302C312E30303000
Over on the Things Network console I could see that exact byte array appear, which converts to an ASCII string:
78.23,42,0.00,1.000
So that’s great! It works.
The Things Network console also has a really nice feature where you can decode your byte array into something more human readable, it’s the Payload Formats tab.
There’s some sample code to extend, so here’s what I ended up with:
function Decoder(bytes, port) {
// Decode an uplink message from a buffer
// (array) of bytes to an object of fields.
var decoded = {};
if (port === 1) {
var stringFromBytes = String.fromCharCode.apply(String, bytes);
var stringArray = stringFromBytes.split(',');
decoded.electrical_conductivity = parseFloat(stringArray[0]);
decoded.total_dissolved_solids = parseFloat(stringArray[1]);
decoded.salinity = parseFloat(stringArray[2]);
decoded.specific_gravity = parseFloat(stringArray[3]);
}
return decoded;
}
It’s very simple, with no error handling or validation (though there is a validation tab when I get some more time).
There’s also a neat Payload Test field, where you can test your decoder as you write it. After I’d ironed out the bugs I switched back over to the Data tab and saw it all working!
Success! The water quality sensor payload is decoded and displayed.
Next step: writing some kind of integrator, so that it can send the data onto a database for visualisation.
It turns out that it wasn’t only the range that might have been causing my connection problems to The Things Network – Andrew Sargent from OpenSensing spotted that in my logs the LoRaWAN RN2903 chip was reporting a firmware version 0.9.5 whereas the version needed to connect is 1.0.3
I’m on macOS, so ideally wanted to use my mac to upgrade the firmware. Here are the steps that I followed:
Upload the PassThrough sketch using the Arduino IDE to The Things Uno so that we can talk directly to the RN2903:
File > Examples > TheThingsNetwork > PassThrough
Back in Terminal, cd into the directory with LoRaDevUtility.jar
Run the Microchip utility: $ /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java -jar LoRaDevUtility.jar
Click ‘RN Module 0’ from the left.
Click the DFU tab.
Upload the new firmware.
Unfortunately clicking the ‘Select File:’ button causes a Java exception on the mac:
Exception in thread "JavaFX Application Thread" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at javafx.fxml.FXMLLoader$MethodHandler.invoke(FXMLLoader.java:1774)
...
Caused by: java.lang.IllegalArgumentException: Folder parameter must be a valid folder
at com.sun.glass.ui.CommonDialogs.convertFolder(CommonDialogs.java:238)
at com.sun.glass.ui.CommonDialogs.showFileChooser(CommonDialogs.java:190)
at com.sun.javafx.tk.quantum.QuantumToolkit.showFileChooser(QuantumToolkit.java:1496)
at javafx.stage.FileChooser.showDialog(FileChooser.java:416)
at javafx.stage.FileChooser.showOpenDialog(FileChooser.java:350)
at fed.FEDFXMLController.onFileBrowse(FEDFXMLController.java:7049)
... 83 more
Decompiling the .jar, I found this in DFUFXMLController.java
@FXML
void onFileBrowse()
{
try {
fileOpenDialog.setInitialDirectory(new File(Preferences.userNodeForPackage(getClass()).get("FilePath", "C:\\")));
fileOpenDialog.getExtensionFilters().add(new javafx.stage.FileChooser.ExtensionFilter(".hex", new String[] { "*.hex" }));
File fileChosen = fileOpenDialog.showOpenDialog(application.getViewer().getScene().getWindow());
if (fileChosen != null) {
selectedFile.setText(fileChosen.getName());
if (application.device.updateValueFlag) {
application.device.dfuPojo.setHexFileName(fileChosen.getAbsolutePath());
}
Preferences.userNodeForPackage(getClass()).put("FilePath", fileChosen.getAbsolutePath().substring(0, fileChosen.getAbsolutePath().indexOf(fileChosen.getName())));
}
} catch (IllegalArgumentException ex) {
if (ex.getMessage().contains("Folder parameter must be a valid folder")) {
fileOpenDialog.setInitialDirectory(new File("C:\\"));
fileOpenDialog.getExtensionFilters().add(new javafx.stage.FileChooser.ExtensionFilter(".hex", new String[] { "*.hex" }));
File fileChosen = fileOpenDialog.showOpenDialog(application.getViewer().getScene().getWindow());
if (fileChosen != null) {
selectedFile.setText(fileChosen.getName());
if (application.device.updateValueFlag) {
application.device.dfuPojo.setHexFileName(fileChosen.getAbsolutePath());
}
Preferences.userNodeForPackage(getClass()).put("FilePath", fileChosen.getAbsolutePath().substring(0, fileChosen.getAbsolutePath().indexOf(fileChosen.getName())));
}
}
}
}
So it looks like C:\\ is hardcoded as the default location, which appears to be why it’s failing unless there’s a default path set (which there isn’t on first run).
This was reported to Microchip in 2016 so is unlikely to be fixed anytime soon.
Leo found a solution for Linux by adding a default FilePath entry, so I used that to add a default entry for macOS:
Open a finder window: $ open ~/Library/Preferences/
Search for com.apple.java.util.prefs in the finder window.
Right-click to open com.apple.java.util.prefs.plist in Xcode
Create a ‘Dictionary’ entry for fed/ and dfu/
Add a ‘String’ entry for FilePath with the value /Users/<your username>/ under fed/ and dfu/
Save and close.
Adding default FilePath values for Java on macOS
Now you should be able to run the Microchip utility and select the firmware to upload: $ /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java -jar LoRaDevUtility.jar
Don’t do what I did and select the unzipped folder, instead copy the file RN2903_Parser.production.unified.hex to the Desktop and select it from there.
If you did try and flash the folder and it failed and won’t connect anymore, fear not – you can select Module > Boot Load Recover. And select the other hex file: RN2903_Parser.production.hex to recover. 😊
Now that I know the sensor works with a standard Arduino Mega, it’s time to try and get it running on The Things Uno.
An Atlas Scientific Conductivity K 1.0 sensor hooked up to The Things Uno
My first attempts getting the Atlas Scientific Uno code running on The Things Uno didn’t get very far. It seemed to be a problem running SoftwareSerial on the default pins 2 & 3.
But when I moved those to pins 10 & 11, I started getting output on the serial monitor! Success.
Next step was to merge that sample code with The Things sample code and see if it all still compiled and ran. More success!
The output from the serial monitor of the Arduino IDE.
The final step is to convert the sensorstring to a byte array so that we can send it to The Things Network.
// Convert sensorstring to a byte array
byte data[sensorstring.length()];
sensorstring.getBytes(data, sizeof(data));
From the TTN docs, the send command to their network is:
// Send the data
ttn.sendBytes(data, sizeof(data));
Finding a spare breadboard, I wired the EZO board to the voltage isolator (which is optional so that any other devices don’t interfere with the conductivity sensor), and then to an Arduino Mega. I thought it better to test it with a plain Arduino before migrating over to The Things Uno, as I haven’t looked into whether the standard serial pins are still usable or not.
Atlas Scientific Conductivity K 1.0 connected to an EZO board, via a power isolation board, to an Arduino Mega.
With everything connected I uploaded the sample code to the Arduino, opened the serial monitor and out came my first data!
EC:0.00
TDS:
SAL:
GRAV:
Noticing that the TDS (total dissolved solids), SAL (salinity) and GRAV (specific gravity) values were all blank, I followed the guide a little longer to see that they’re all off by default. Setting these to return values is as simple as sending the following commands via the Arduino IDE serial monitor:
// Turn on TDS (capital letter O, capital letters TDS, number 1, separated by commas)
O,TDS,1
// Turn on SAL
O,S,1
// Turn on GRAV
O,SG,1
Once you send a command you should see the return value: *OK
Then the output will look more like this:
EC:0.00
TDS:0
SAL:0.00
GRAV:1.000
Next step is to calibrate the sensor for dry air. To do this, send the command using the Arduino IDE serial monitor:
Cal,dry
The other calibration steps you can do are to check against the supplied samples. My samples are 12,880 uS/cm and 80,000 uS/cm which can be set to low and high calibration settings using the commands:
// Put the sensor in the 12,880 sample, and then send the command:
Cal,low,12880
EC:13580
TDS:7336
SAL:7.83
GRAV:1.007
*OK
EC:13580
TDS:7337
SAL:7.83
GRAV:1.007
// Put the sensor in the 80,000 sample, and then send the command:
Cal,high,80000
EC:82530
TDS:44569
SAL:42.00
GRAV:1.041
*OK
EC:79990
TDS:43195
SAL:42.00
GRAV:1.040
To save the calibration from your device, send:
// Send the export calibration command
Export
// My response
EC:53D44200803F
// If ever you need to import it again, simply type
Import
Now we’re ready for our first water quality sample! I poured a glass of Port Willunga tap water in the 80’s cottage I was staying at, and it gave these values:
So it works! But what do those values mean and how do they compare to reported averages?
From page 56 of this 2013/14 SA Water drinking quality report, here are some water quality Total Dissolved Solids figures from areas nearby in mg/L (which is the same as ppm reported by the sensor):
Town
Min TDS
Max TDS
Ave TDS
Myponga
320 mg/L
410 mg/L
353 mg/L
Mount Compass
120 mg/L
260 mg/L
172 mg/L
Port Willunga’s average that I read of 326 mg/L (only a few readings on one day) sits towards the Myponga end of those two readings, which seems to validate the data I just read.
Just out of interest, on page 54 of that report, the “Aesthetic guideline” for water TDS is less than or equal to 600 mg/L. So I think that’s a safe upper limit to compare against for drinking water.
Update: I found that the SA Water “what’s in your water” search returned this 2018 data from South Metro: Average 290 mg/L. So that’s in the same ballpark too which is nice.
Next step is to get it working with I2C on The Things Uno, so that I can push the data to their network.
I might also build a quick iOS Bluetooth app and plug it into my Red Bear BLE Nano so that I can send data to my phone too.
Update: Today I tested some salt water from the beach at Port Willunga 2nd Jan 2019:
// Tap water at an apartment block 2nd Jan 2019
EC:600.4
TDS:324
SAL:0.29
GRAV:1.000
// Tap water at a second apartment block 4th Jan 2019
EC:556.3
TDS:300
SAL:0.27
GRAV:1.000
// Pura tap (2 year old filter)
EC:567.7
TDS:307
SAL:0.28
GRAV:1.000
// Pura tap (6 month old filter)
EC:530.5
TDS:286
SAL:0.26
GRAV:1.000
An average reading from Saint Kilda East in Melbourne:
// Tap water from St Kilda East 6th Jan 2019
EC:61.37
TDS:33
SAL:0.00
GRAV:1.000
// Rain water from St Kilda East 6th Jan 2019
EC:38.91
TDS:21
SAL:0.00
GRAV:1.000
An average reading from a Melbourne CBD apartment (top of the CBD):
// Tap water from Melbourne CBD apartment 10th Jan 2019
EC:117.9
TDS:64
SAL:0.00
GRAV:1.000
Spring water:
// Neverfail spring water (box)
EC:211.6
TDS:114
SAL:0.10
GRAV:1.000
// Mount Franklin spring water (600mL bottle)
EC:162.9
TDS:88
SAL:0.00
GRAV:1.000
// Cool Ridge spring water (600mL bottle)
EC:79.76
TDS:43
SAL:0.00
GRAV:1.000
Out of the box The Things Uno has all the code on it to get started. All I needed to do was plug it into my laptop and open a serial monitor in the Arduino IDE at 9600 baud.
The Things Uno – out of the box it Just Works™️
Green lights flashed, and its status printed to the console. Happy days.
I followed the Uno Quick Start guide (which is really great), updated my Arduino IDE, added TheThingsNetwork library, setup my Application, added my device to the application, copied the appEui & appKey to my example sketch and uploaded it.
Everything looked happy, but then 😔
-- STATUS
EUI: <eui>
Battery: 3253
AppEUI: 0000000000000000
DevEUI: <devEui>
Data Rate: 3
RX Delay 1: 1000
RX Delay 2: 2000
-- JOIN
Model: RN2903
Version: 0.9.5
Sending: mac set deveui <devEui>
Sending: mac set adr off
Sending: mac set deveui <devEui>
Sending: mac set appeui <appID>
Sending: mac set appkey <secret>
Sending: mac save
<snip>
Sending: mac set pwridx 5
Sending: mac set retx 7
Sending: mac set dr 3
Sending: mac join otaa
Join not accepted: denied
Check your coverage, keys and backend status.
Join not accepted: denied. Check your coverage, keys and backend status.
Double checking the sample code and keys didn’t find any bugs, so it was time to look at The Things Network Mapper to see if I was actually in range here at Port Willunga.
December 2018, Port Willunga is out of range.
Bummer. ~22.8km away from the nearest gateway, and the theoretical range of LoRaWAN is ~15km.
So I’ll take a break from connecting to the network and get back to getting The Things Uno talking to the Atlas Scientific Water Conductivity sensor. Then when that’s all sorted, I’ll take a little drive down the coast and connect that way.
I really love the setup process of The Things Network – straight forward guide, and sample code that all works perfectly. 😊
Update: Once I reached Adelaide CBD I plugged in The Things Uno again in the hope it would connect to the many close gateways… but unfortunately I had the same problem.
A quick chat in the #Adelaide channel on thethingsnetwork.slack.com and Andrew noticed that my Uno’s LoRaWAN firmware on the Microchip RN2903 is running old firmware (Version 0.9.5), and apparently it needs to be running Version 1.0.3 to connect.
Six months ago I moved to Williamstown Beach just outside of Melbourne. It’s a nice beachside town about 15km from the centre of the city. The beach is a couple of blocks away via a walk through the botanic gardens – a pretty ideal place to replenish after long days in front of computers (my day jobs include being a software developer for ACMI and New Internationalist).
The beach here has a really diverse crowd – many countries of Africa are represented, as well as a large portion of southern Europe and Asia. It’s a really popular place for both families with young kids and pets, as well as groups of teenage and 20-something friends.
But there is one blot on this otherwise idyllic bay, and that’s a big waste water drain that exits right next to the swimming beach. A waste water drain wouldn’t be so much of a big deal on its own, but Williamstown’s proximity to a large shipping port, industrial area, and oil refinery made me curious to see whether the output of it is at all tainted.
That curiosity peaked after a recent factory fire and subsequent rains forced the local council to put up signs warning that the water was unfit to swim in or even for pets to play in.
EPA Victoria has a twitter account that tweets water quality for beaches, but it doesn’t seem to have an entry for Williamstown and some of the alerts are based on historical data.
So I decided to start building my own monitoring system, to see what the cost would be to setup alerts of my own.
My first stop was the Public Lab Water Quality Sensor project. Being five years old there are a lot of broken links, but there was one that was useful – the link to Atlas Scientific sensors.
Given the cost of Atlas sensors (~AUD$415 each delivered), I figured for my first tests I might choose one and see how it goes. Then buy more if that one ends up being useful.
$400 is about double my comfortable limit for hobby projects, so I put the feelers out on the Hackerspace Adelaide list & Twitter to see if anyone wanted to share the cost and the sensors.
I’m wanting to do some basic water quality analysis of the tap water and beach water around Australia. This Arduino/Raspberry Pi compatible water conductivity kit is ~AUD$415 delivered. Is anyone keen to chip in to share use? Data would be public. https://t.co/YpQRTwW1Tg
Almost instantly, Jeannine very kindly put me onto Leo and The Things Network, who not only got TTN to donate me a The Things Uno with LoRaWAN (Low power, wide area network with ~15km range), but also agreed to sponsor me $100 towards the costs of the sensors (THANK YOU!).
The TPS would probably work nicely (The Bluelab is seemingly a standalone reader).
But seeing I wanted to open this up as public data, and have the possibility of an easily replicated solution, I decided to go with off-the-shelf products that have great documentation, Arduino sample code, wiring diagram and calibration samples (seriously, how nice is their documentation!!!).
The great unboxing.
Atlas Scientific and Leo from The Things Network were super prompt with their deliveries, they were each really nicely packaged and arrived within a week – a nice start to any project.
