Water Quality iOS app

LoRa is great when there’s a network available, but for in-person water quality readings I thought it’d be convenient to add Bluetooth.

I had a spare BLE Nano v2 sitting around, so decided to wire that in. It acts as a relay, broadcasting everything that the Water Quality sensor outputs over serial, and pushing it out via Bluetooth.

Because this is the same setup as my air quality sensor project, I could clone the Air Quality iOS app to become a Water Quality iOS app. The app saves all of the data to my iPhone/iPad’s Core Database, and there’s an export to CSV function too.

Quick graph of the first sensor data I measured.

Sample data: Google Spreadsheet / Raw CSV.

It’s pretty snug on the mini breadboard, and I cheated a bit by adding in the 3.3v BLE board on the wrong side of the power isolator to avoid having to drop down the 5v TX pin from the Arduino side to 3.3v, but it all still fits in the waterproof enclosure and functions very nicely.

The Bluetooth BLE board is the red one on the right.

Next step is to update my Rails site to be able to include water quality data, add some API authentication, and host it. Then all of the data the iOS apps are getting will be posted to a database and graphed nicely, both on a linear graph, and on a map.

Debugging LoRa connectivity

After successfully upgrading the firmware of my Uno’s RN2903 LoRa chip, I was still having trouble connecting to The Things Network and seeing a lot of these lines in the serial monitor:

Sending: mac join otaa 
Join not accepted: denied
Check your coverage, keys and backend status.

I’d tried five different locations across two states, so was getting pretty paranoid that it was either something I’d done in my Arduino water quality sensor reading code (I was using Software Serial).

So I asked Leo from TTN Adelaide if it might be possible for him to send me a known working Uno so that I could try it here using stock software. His setup was identical to the one I had been trying, but at least we could rule out hardware failure, and it’d tell me a lot more about trusting my code.

As soon as it arrived I plugged it into my USB power brick so that my laptop couldn’t do anything weird to it (super paranoid), but it didn’t connect successfully. So after a couple of minutes I plugged it into my laptop so I could read the serial monitor output. Same output as mine.

So at this point I could at least feel a little better about my code and hardware, but it was still a bit of a mystery why all of the locations I’d tried didn’t work.

I wandered around the Melbourne CBD with it switched on, monitoring TTN console on my phone for uploads… still nothing.

In a last ditch effort to find some signal, I got myself onto a 38th floor rooftop, switched it on, and almost instantly the join confirmation and payloads started streaming in.

So it was just blackspots all over the Melbourne CBD, East St Kilda, Southbank, Adelaide CBD and Port Willunga.

Test the payload

Next thing to do was to test the payload to see if I was sending a readable byte array.

-- SENSOR READING
EC:78.23
TDS:42
SAL:0.00
GRAV:1.000


-- BYTE ARRAY TO SEND
37 38 2E 32 33 2C 34 32 2C 30 2E 30 30 2C 31 2E 30 30 30 00 

Sending: mac tx uncnf 1 37382E32332C34322C302E30302C312E30303000

Over on the Things Network console I could see that exact byte array appear, which converts to an ASCII string:

78.23,42,0.00,1.000

So that’s great! It works.

The Things Network console also has a really nice feature where you can decode your byte array into something more human readable, it’s the Payload Formats tab.

There’s some sample code to extend, so here’s what I ended up with:

function Decoder(bytes, port) {
  // Decode an uplink message from a buffer
  // (array) of bytes to an object of fields.
  var decoded = {};

  if (port === 1) {
    var stringFromBytes = String.fromCharCode.apply(String, bytes);
    var stringArray = stringFromBytes.split(',');
    decoded.electrical_conductivity = parseFloat(stringArray[0]);
    decoded.total_dissolved_solids = parseFloat(stringArray[1]);
    decoded.salinity = parseFloat(stringArray[2]);
    decoded.specific_gravity = parseFloat(stringArray[3]);
  }

  return decoded;
}

It’s very simple, with no error handling or validation (though there is a validation tab when I get some more time).

There’s also a neat Payload Test field, where you can test your decoder as you write it. After I’d ironed out the bugs I switched back over to the Data tab and saw it all working!

Success! The water quality sensor payload is decoded and displayed.

Next step: writing some kind of integrator, so that it can send the data onto a database for visualisation.

Upgrading The Things Uno RN2903 firmware

It turns out that it wasn’t only the range that might have been causing my connection problems to The Things Network – Andrew Sargent from OpenSensing spotted that in my logs the LoRaWAN RN2903 chip was reporting a firmware version 0.9.5 whereas the version needed to connect is 1.0.3

I’m on macOS, so ideally wanted to use my mac to upgrade the firmware. Here are the steps that I followed:

  • Download the Microchip Development Suite software and install just the Application: https://www.microchip.com/DevelopmentTools/ProductDetails/DV164140-2
  • Install Java8 to run it via Homebrew: $ brew cask install java8
    • You may also need to tap versions first if you haven’t already: $ brew tap caskroom/versions
  • Find that java8 installation (I also have a new version of Java I didn’t want to disrupt): $ /usr/libexec/java_home -v 1.8
  • Download the new RN2903 SA1.0.3 firmware (Hex) from: https://github.com/TheThingsNetwork/arduino-device-lib/files/1899025/RN2903.SA1.0.3.Hex.zip
  • Upload the PassThrough sketch using the Arduino IDE to The Things Uno so that we can talk directly to the RN2903:
  • File > Examples > TheThingsNetwork > PassThrough
  • Back in Terminal, cd into the directory with LoRaDevUtility.jar
  • Run the Microchip utility: $ /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java -jar LoRaDevUtility.jar
  • Click ‘RN Module 0’ from the left.
  • Click the DFU tab.
  • Upload the new firmware.

Unfortunately clicking the ‘Select File:’ button causes a Java exception on the mac:

Exception in thread "JavaFX Application Thread" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at javafx.fxml.FXMLLoader$MethodHandler.invoke(FXMLLoader.java:1774)
...
Caused by: java.lang.IllegalArgumentException: Folder parameter must be a valid folder
	at com.sun.glass.ui.CommonDialogs.convertFolder(CommonDialogs.java:238)
	at com.sun.glass.ui.CommonDialogs.showFileChooser(CommonDialogs.java:190)
	at com.sun.javafx.tk.quantum.QuantumToolkit.showFileChooser(QuantumToolkit.java:1496)
	at javafx.stage.FileChooser.showDialog(FileChooser.java:416)
	at javafx.stage.FileChooser.showOpenDialog(FileChooser.java:350)
	at fed.FEDFXMLController.onFileBrowse(FEDFXMLController.java:7049)
	... 83 more

Decompiling the .jar, I found this in DFUFXMLController.java

@FXML
  void onFileBrowse()
  {
    try {
      fileOpenDialog.setInitialDirectory(new File(Preferences.userNodeForPackage(getClass()).get("FilePath", "C:\\")));
      fileOpenDialog.getExtensionFilters().add(new javafx.stage.FileChooser.ExtensionFilter(".hex", new String[] { "*.hex" }));
      File fileChosen = fileOpenDialog.showOpenDialog(application.getViewer().getScene().getWindow());
      
      if (fileChosen != null) {
        selectedFile.setText(fileChosen.getName());
        if (application.device.updateValueFlag) {
          application.device.dfuPojo.setHexFileName(fileChosen.getAbsolutePath());
        }
        Preferences.userNodeForPackage(getClass()).put("FilePath", fileChosen.getAbsolutePath().substring(0, fileChosen.getAbsolutePath().indexOf(fileChosen.getName())));
      }
    } catch (IllegalArgumentException ex) {
      if (ex.getMessage().contains("Folder parameter must be a valid folder")) {
        fileOpenDialog.setInitialDirectory(new File("C:\\"));
        fileOpenDialog.getExtensionFilters().add(new javafx.stage.FileChooser.ExtensionFilter(".hex", new String[] { "*.hex" }));
        File fileChosen = fileOpenDialog.showOpenDialog(application.getViewer().getScene().getWindow());
        
        if (fileChosen != null) {
          selectedFile.setText(fileChosen.getName());
          if (application.device.updateValueFlag) {
            application.device.dfuPojo.setHexFileName(fileChosen.getAbsolutePath());
          }
          Preferences.userNodeForPackage(getClass()).put("FilePath", fileChosen.getAbsolutePath().substring(0, fileChosen.getAbsolutePath().indexOf(fileChosen.getName())));
        }
      }
    }
  }

So it looks like C:\\ is hardcoded as the default location, which appears to be why it’s failing unless there’s a default path set (which there isn’t on first run).

This was reported to Microchip in 2016 so is unlikely to be fixed anytime soon.

Leo found a solution for Linux by adding a default FilePath entry, so I used that to add a default entry for macOS:

  • Open a finder window: $ open ~/Library/Preferences/
  • Search for com.apple.java.util.prefs in the finder window.
  • Right-click to open com.apple.java.util.prefs.plist in Xcode
  • Create a ‘Dictionary’ entry for fed/ and dfu/
  • Add a ‘String’ entry for FilePath with the value /Users/<your username>/ under fed/ and dfu/
  • Save and close.
Adding default FilePath values for Java on macOS

Now you should be able to run the Microchip utility and select the firmware to upload: $ /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/bin/java -jar LoRaDevUtility.jar

Don’t do what I did and select the unzipped folder, instead copy the file RN2903_Parser.production.unified.hex to the Desktop and select it from there.

If you did try and flash the folder and it failed and won’t connect anymore, fear not – you can select Module > Boot Load Recover. And select the other hex file: RN2903_Parser.production.hex to recover. 😊

Atlas Conductivity K 1.0 to TTN Uno

Now that I know the sensor works with a standard Arduino Mega, it’s time to try and get it running on The Things Uno.

An Atlas Scientific Conductivity K 1.0 sensor hooked up to The Things Uno

My first attempts getting the Atlas Scientific Uno code running on The Things Uno didn’t get very far. It seemed to be a problem running SoftwareSerial on the default pins 2 & 3.

But when I moved those to pins 10 & 11, I started getting output on the serial monitor! Success.

Next step was to merge that sample code with The Things sample code and see if it all still compiled and ran. More success!

The output from the serial monitor of the Arduino IDE.

The final step is to convert the sensorstring to a byte array so that we can send it to The Things Network.

// Convert sensorstring to a byte array
byte data[sensorstring.length()];
sensorstring.getBytes(data, sizeof(data));

From the TTN docs, the send command to their network is:

// Send the data
ttn.sendBytes(data, sizeof(data));

The complete code is now all up in this GitHub repo:
github.com/sighmon/water-quality-sensors

Exciting!

Atlas K 1.0 water conductivity sensor

Connecting an Atlas Scientific Conductivity K 1.0 sensor to an Arduino.

First of all, let me just say that I adore the packaging of the EZO conductivity and power isolation circuit boards.

They arrived in two adorable little cryogenic chambers.

They were a lot smaller than I expected, which is a nice surprise, as it’ll mean a smaller end product once I’ve packaged them all into a box.

The EZO Conductivity Circuit Datasheet is more like a getting started guide, so I followed that first.

Finding a spare breadboard, I wired the EZO board to the voltage isolator (which is optional so that any other devices don’t interfere with the conductivity sensor), and then to an Arduino Mega. I thought it better to test it with a plain Arduino before migrating over to The Things Uno, as I haven’t looked into whether the standard serial pins are still usable or not.

Wiring diagram without the power isolator.
Wiring diagram with the power isolator.
Atlas Scientific Conductivity K 1.0 connected to an EZO board, via a power isolation board, to an Arduino Mega.

With everything connected I uploaded the sample code to the Arduino, opened the serial monitor and out came my first data!

EC:0.00
TDS:
SAL:
GRAV:

Noticing that the TDS (total dissolved solids), SAL (salinity) and GRAV (specific gravity) values were all blank, I followed the guide a little longer to see that they’re all off by default. Setting these to return values is as simple as sending the following commands via the Arduino IDE serial monitor:

// Turn on TDS (capital letter O, capital letters TDS, number 1, separated by commas)
O,TDS,1

// Turn on SAL
O,S,1

// Turn on GRAV
O,SG,1

Once you send a command you should see the return value: *OK

Then the output will look more like this:

EC:0.00
TDS:0
SAL:0.00
GRAV:1.000

Next step is to calibrate the sensor for dry air. To do this, send the command using the Arduino IDE serial monitor:

Cal,dry

The other calibration steps you can do are to check against the supplied samples. My samples are 12,880 uS/cm and 80,000 uS/cm which can be set to low and high calibration settings using the commands:

// Put the sensor in the 12,880 sample, and then send the command:
Cal,low,12880

EC:13580
TDS:7336
SAL:7.83
GRAV:1.007

*OK
EC:13580
TDS:7337
SAL:7.83
GRAV:1.007

// Put the sensor in the 80,000 sample, and then send the command:
Cal,high,80000

EC:82530
TDS:44569
SAL:42.00
GRAV:1.041

*OK
EC:79990
TDS:43195
SAL:42.00
GRAV:1.040

To save the calibration from your device, send:

// Send the export calibration command
Export

// My response
EC:53D44200803F

// If ever you need to import it again, simply type
Import

Now we’re ready for our first water quality sample! I poured a glass of Port Willunga tap water in the 80’s cottage I was staying at, and it gave these values:

EC:601.2
TDS:325
SAL:0.29
GRAV:1.000

EC:602.0
TDS:325
SAL:0.29
GRAV:1.000

EC:602.6
TDS:325
SAL:0.29
GRAV:1.000

EC:602.8
TDS:326
SAL:0.29
GRAV:1.000

EC:603.4
TDS:326
SAL:0.29
GRAV:1.000

EC:603.8
TDS:326
SAL:0.29
GRAV:1.000

EC:604.0
TDS:326
SAL:0.29
GRAV:1.000

EC:604.4
TDS:326
SAL:0.29
GRAV:1.000

EC:604.3
TDS:326
SAL:0.29
GRAV:1.000

So it works! But what do those values mean and how do they compare to reported averages?

From page 56 of this 2013/14 SA Water drinking quality report, here are some water quality Total Dissolved Solids figures from areas nearby in mg/L (which is the same as ppm reported by the sensor):

TownMin TDSMax TDSAve TDS
Myponga320 mg/L410 mg/L353 mg/L
Mount Compass120 mg/L260 mg/L172 mg/L

Port Willunga’s average that I read of 326 mg/L (only a few readings on one day) sits towards the Myponga end of those two readings, which seems to validate the data I just read.

Just out of interest, on page 54 of that report, the “Aesthetic guideline” for water TDS is less than or equal to 600 mg/L. So I think that’s a safe upper limit to compare against for drinking water.

Update: I found that the SA Water “what’s in your water” search returned this 2018 data from South Metro: Average 290 mg/L. So that’s in the same ballpark too which is nice.

Next step is to get it working with I2C on The Things Uno, so that I can push the data to their network.

I might also build a quick iOS Bluetooth app and plug it into my Red Bear BLE Nano so that I can send data to my phone too.

Update: Today I tested some salt water from the beach at Port Willunga 2nd Jan 2019:

-- SENSOR READING
EC:51291
TDS:27697
SAL:33.69
GRAV:1.026

-- SENSOR READING
EC:51341
TDS:27724
SAL:33.73
GRAV:1.026

-- SENSOR READING
EC:51381
TDS:27750
SAL:33.76
GRAV:1.026

-- SENSOR READING
EC:51431
TDS:27775
SAL:33.79
GRAV:1.026

-- SENSOR READING
EC:51481
TDS:27799
SAL:33.83
GRAV:1.026

-- SENSOR READING
EC:51501
TDS:27814
SAL:33.85
GRAV:1.026

-- SENSOR READING
EC:51521
TDS:27826
SAL:33.86
GRAV:1.026

-- SENSOR READING
EC:51561
TDS:27843
SAL:33.89
GRAV:1.026

-- SENSOR READING
EC:51581
TDS:27857
SAL:33.91
GRAV:1.026

-- SENSOR READING
EC:51631
TDS:27881
SAL:33.94
GRAV:1.026

And an average reading from the Adelaide CBD:

// Tap water at an apartment block 2nd Jan 2019
EC:600.4
TDS:324
SAL:0.29
GRAV:1.000

// Tap water at a second apartment block 4th Jan 2019
EC:556.3
TDS:300
SAL:0.27
GRAV:1.000

// Pura tap (2 year old filter)
EC:567.7
TDS:307
SAL:0.28
GRAV:1.000

// Pura tap (6 month old filter)
EC:530.5
TDS:286
SAL:0.26
GRAV:1.000

An average reading from Saint Kilda East in Melbourne:

// Tap water from St Kilda East 6th Jan 2019
EC:61.37
TDS:33
SAL:0.00
GRAV:1.000

// Rain water from St Kilda East 6th Jan 2019
EC:38.91
TDS:21
SAL:0.00
GRAV:1.000

An average reading from a Melbourne CBD apartment (top of the CBD):

// Tap water from Melbourne CBD apartment 10th Jan 2019
EC:117.9
TDS:64
SAL:0.00
GRAV:1.000

Spring water:

// Neverfail spring water (box)
EC:211.6
TDS:114
SAL:0.10
GRAV:1.000

// Mount Franklin spring water (600mL bottle)
EC:162.9
TDS:88
SAL:0.00
GRAV:1.000

// Cool Ridge spring water (600mL bottle)
EC:79.76
TDS:43
SAL:0.00
GRAV:1.000

Connecting to The Things Network

Out of the box The Things Uno has all the code on it to get started. All I needed to do was plug it into my laptop and open a serial monitor in the Arduino IDE at 9600 baud.

The Things Uno – out of the box it Just Works™️

Green lights flashed, and its status printed to the console. Happy days.

-- STATUS
EUI: <eui>
Battery: 3253
AppEUI: 0000000000000000
DevEUI: <devEui>
Data Rate: 3
RX Delay 1: 1000
RX Delay 2: 2000

I followed the Uno Quick Start guide (which is really great), updated my Arduino IDE, added TheThingsNetwork library, setup my Application, added my device to the application, copied the appEui & appKey to my example sketch and uploaded it.

Everything looked happy, but then 😔

-- STATUS
EUI: <eui>
Battery: 3253
AppEUI: 0000000000000000
DevEUI: <devEui>
Data Rate: 3
RX Delay 1: 1000
RX Delay 2: 2000
-- JOIN
Model: RN2903
Version: 0.9.5
Sending: mac set deveui <devEui>
Sending: mac set adr off
Sending: mac set deveui <devEui>
Sending: mac set appeui <appID>
Sending: mac set appkey <secret>
Sending: mac save 
<snip>
Sending: mac set pwridx 5
Sending: mac set retx 7
Sending: mac set dr 3
Sending: mac join otaa 
Join not accepted: denied
Check your coverage, keys and backend status.

Join not accepted: denied. Check your coverage, keys and backend status.

Double checking the sample code and keys didn’t find any bugs, so it was time to look at The Things Network Mapper to see if I was actually in range here at Port Willunga.

December 2018, Port Willunga is out of range.

Bummer. ~22.8km away from the nearest gateway, and the theoretical range of LoRaWAN is ~15km.

So I’ll take a break from connecting to the network and get back to getting The Things Uno talking to the Atlas Scientific Water Conductivity sensor. Then when that’s all sorted, I’ll take a little drive down the coast and connect that way.

I really love the setup process of The Things Network – straight forward guide, and sample code that all works perfectly. 😊

Update: Once I reached Adelaide CBD I plugged in The Things Uno again in the hope it would connect to the many close gateways… but unfortunately I had the same problem.

A quick chat in the #Adelaide channel on thethingsnetwork.slack.com and Andrew noticed that my Uno’s LoRaWAN firmware on the Microchip RN2903 is running old firmware (Version 0.9.5), and apparently it needs to be running Version 1.0.3 to connect.

Follow along as I grapple with upgrading the firmware

Water quality monitor

Six months ago I moved to Williamstown Beach just outside of Melbourne. It’s a nice beachside town about 15km from the centre of the city. The beach is a couple of blocks away via a walk through the botanic gardens – a pretty ideal place to replenish after long days in front of computers (my day jobs include being a software developer for ACMI and New Internationalist).

The beach here has a really diverse crowd – many countries of Africa are represented, as well as a large portion of southern Europe and Asia. It’s a really popular place for both families with young kids and pets, as well as groups of teenage and 20-something friends.

But there is one blot on this otherwise idyllic bay, and that’s a big waste water drain that exits right next to the swimming beach. A waste water drain wouldn’t be so much of a big deal on its own, but Williamstown’s proximity to a large shipping port, industrial area, and oil refinery made me curious to see whether the output of it is at all tainted.

That curiosity peaked after a recent factory fire and subsequent rains forced the local council to put up signs warning that the water was unfit to swim in or even for pets to play in.

EPA Victoria has a twitter account that tweets water quality for beaches, but it doesn’t seem to have an entry for Williamstown and some of the alerts are based on historical data.

So I decided to start building my own monitoring system, to see what the cost would be to setup alerts of my own.

My first stop was the Public Lab Water Quality Sensor project. Being five years old there are a lot of broken links, but there was one that was useful – the link to Atlas Scientific sensors.

I also read up about the different water quality parameters you can measure:

  • Temperature
  • pH
  • Salinity (Electrical Conductivity)
  • Turbidity
  • Suspended solids
  • Dissolved oxygen
  • Heavy metals
  • Nutrients

Given the cost of Atlas sensors (~AUD$415 each delivered), I figured for my first tests I might choose one and see how it goes. Then buy more if that one ends up being useful.

So with that in mind, I decided on the Conductivity K 1.0 kit.

Atlas Scientific Conductivity K 1.0 kit

$400 is about double my comfortable limit for hobby projects, so I put the feelers out on the Hackerspace Adelaide list & Twitter to see if anyone wanted to share the cost and the sensors.

Almost instantly, Jeannine very kindly put me onto Leo and The Things Network, who not only got TTN to donate me a The Things Uno with LoRaWAN (Low power, wide area network with ~15km range), but also agreed to sponsor me $100 towards the costs of the sensors (THANK YOU!).

Big shout out to Tisham Dhar of Whatnick fame too, who also offered to help out (Tisham makes open hardware energy monitors that are awesome).

The Hackerspace Adelaide community also came through with some low cost solutions:

The TPS would probably work nicely (The Bluelab is seemingly a standalone reader).

But seeing I wanted to open this up as public data, and have the possibility of an easily replicated solution, I decided to go with off-the-shelf products that have great documentation, Arduino sample code, wiring diagram and calibration samples (seriously, how nice is their documentation!!!).

The great unboxing.

Atlas Scientific and Leo from The Things Network were super prompt with their deliveries, they were each really nicely packaged and arrived within a week – a nice start to any project.

Next steps: