Blog


A Time Triggered Action!

This example shows how to take an action based on a time or schedule. It uses the excellent WP Control plugin.

This plugin allows you to easily set the time window or repeating schedule that you want to use. Here is my logic that I would like to implement. I’ve highlighted my criteria in green and my action in red

If it is 6pm I want to start watering my plants for 3 minutes.

Firstly, I set up a new cron schedule using WP Control. This looks like the picture below. Choose a daily schedule and don’t forget when you first want it to run! I can now start writing my logic in the box.

WP Control – Cron Event

In order to turn on my pump I send a mysensors_in/10/6/1/0/2 the playload contains the number of minutes that the pump should be on.

So lets have a first go at implementing our logic. It is possible to send data to your Mqtt broker really easily using the MqttCogs_Plugin class. This is REALLY easy. The sendMqtt function will even automatically buffer messages for MySensors nodes that implement smartSleep.

I’ve expanded the call parameters into variables but this is not necessary.

$clientid = 'cron_w'; //can be any short text
$outtopic = 'mysensors_in/10/6/1/0/2'; //topic to write to
$outpayload = 3; //payload, on for 3 minutes
$qos = 1; //qos
$retain = 0; //retain
$error = new stdClass();

$mqttcogs = new MqttCogs_Plugin();
$mqttcogs->sendMqtt($clientid, $outtopic, $outpayload, $qos, $retain, $error);

Here’s a second go. This one picks up the return from the function and then emails the result. I haven’t expanded the variables this time to keep things concise.

$mqttcogs = new MqttCogs_Plugin();
$error = new stdClass();

//note that this function returns
// true (message ok or buffered) or 
// false for error
// I just use the $error object though...
$mqttcogs->sendMqtt('cron_w', 'mysensors_in/10/6/1/0/2', 3, 1,0, $error); 
	switch($error->status) {
		case 'ok':
			wp_mail( 'test@test.com', 'Watering on', 'Message sent' ); 
		break;
		case 'buffered':
			wp_mail( 'test@test.com', 'Watering on', 'Message buffered' ); 
		break;
		case 'error':
			wp_mail( 'test@test.com', 'Watering FAIL', $error->errors[0].reason); 	
		break;
	}




Visualize OwnTracks GPS Data

This post describes how to visualize Mqtt data from the OwnTracks App.

For this experiment I installed OwnTracks onto my phone and filled in my credentials for my Mqtt broker.

Problem 1: My Mqtt broker uses port 14027. Outgoing data to this port wasn’t allowed on the particular Wifi network I was using. This took me ages to work out and was solved simply by turning off Wifi and using the phone data connection.

When this happens OwnTracks won’t fall back and use a different data connection so you might get delayed tracking data when a user phone moves in and out of particular Wifi networks.

OwnTracks topic and mqtt message format

The OwnTracks format is well documented, see here. It is a little verbose but it does contain longitude, latitude and speed. Be aware that the data can change depending on your phone platform. The topic is in the format owntracks/user/device. Both user and device can be set in the App.

Here’s an example of the two types of packets that I received from my Android device :

{	"_type":"location",
	"acc":8,
	"alt":56,
	"batt":75,
	"conn":"m",
	"lat":53.8165549,
	"lon":-1.6013796,
	"tid":"1",
	"tst":1557255979,
	"vac":8,
	"vel":0
}

{     "_type":"lwt",
     "tst":1557286504
}

The lwt type is a special ‘last will and testament’ type. It is sent when the device disconnects from the Mqtt broker. This can happen often, particularly if you are travelling!

Problem 2: I need to combine the deviceid from the topic and the lng/lats and timestamp from the payload to get a meaningful piece of data to plot.

Problem 3: What do I do with the lwt (last will and testament) message? This tells me when the device disconnected, although I can’t find a corresponding ‘device has connected’ message.

I can change or shape the data by using a mqttcogs_msg_in_pre filter hook. This is called just before the data is persisted to the wordpress database. I’m going to do aim to do the following:

  • Ignore messages that aren’t of type ‘location’
  • Use the GPS datetime rather than the server time in the utc field
  • Add a new field ‘Desc’ that I actually never use. This is simply to demonstrate how the payload could be manipulated.

add_filter('mqttcogs_msg_in_pre', 'mqttcogs_msg_in_pre', 10, 3);
 function mqttcogs_msg_in_pre($publish_object, $datetime)
 {
     $topic = $publish_object->getTopic();
     //filter by topic
     if (!(substr( $topic, 0, 9 ) === "owntracks")) {
         return $publish_object;
     }
     
     $mqttmsg = $publish_object->getMessage();
     $payload = json_decode($mqttmsg);

     //only do this to correct type of messages
     if ($payload->_type != 'location') {
        return NULL;
     }
     
     //update utc time to GPS datetime
     if (property_exists($payload, 'tst')) {
        $datetime->setTimestamp((int) $payload->tst);
     }
     
     //Now manipulate the payload. This extra attribute isn't actually 
     //used!
     $speed = $payload->vel;
     $nicedatetime = $datetime->format('Y-m-d H:i:s');
     $payload->Desc  = 'Speed:'.$speed.'kph At:'.$nicedatetime;
     $publish_object->setMessage(json_encode($payload));
     return $publish_object;
 }

….and here is the result. I’ve included the shortcode below. Remember that as of version 2.2 the longitude and latitude will be automatically extracted for the Map Visualization. Map looks good but the Google Map Visualization isn’t set up to refresh very nicely. When you set the refresh_secs=”60″ and the data is refreshed, this particular visualization draws the Map tiles and resets the zoom level. Not very useful!

[ mqttcogs_drawgoogle 
	refresh_secs="120" 
	charttype="Map" 
	options="{height:300,zoomLevel:12,showTooltip: true,showInfoWindow:true,showLine:true}"]
	[ mqttcogs_data 
		limit="1"
		order="DESC"
		topics="owntracks/another/ckandroid"]
[ /mqttcogs_drawgoogle]




Playing with GPS Data using Filter Hooks

I wrote a really basic example here that showed how to plot some GPS points onto map. The example works well but you may have noticed that it relies on the data in the Mqtt message payload formatted in a particular way. Remember what it looked like?

If you’ve worked with GPS devices before you know that the GPS data that ends up at the Mqtt broker won’t look like that. That’s far too simple!

MqttCogs provides a Filter Hook to allow you change the incoming payload before it is persisted to the database. This allows you to reshape the JSON in the payload to a form that can be used by the MqttCogs shortcodes.

To use the hook you need to be able to write some php. I used My Custom Functions wordpress plugin. This allows you to quickly add some php to your site without messing around with functions.php or any other part of your site. I highly recommend this plugin.

My code looks something like this. It reads the payload, changes the format of the incoming payload data. The data is then persisted as normal.

add_filter('mqttcogs_msg_in_pre', 'mqttcogs_msg_in_pre', 10, 3);
 function mqttcogs_msg_in_pre($publish_object, $utc)
 {
     //do nothing if we are not interested in the topic
     if ($publish_object.getTopic() != 'sometopicthatcontainslnglat") {
           return;
     }
     
     //reshape the JSON data
     $payload = $publish_object.getMessage();
     $payload = reshapethedatafunction($payload);
     $publish_object.setMessage($payload);
 }

To extract and plot the values from my json payload I need to instruct MqttCogs where to get them from. Have a read of visualizing mqtt json payload this provides a worked example of how to show your lng lats on a map.

Check out Visualizing OwnTracks GPS Data this uses this principle to show owntracks GPS data on a map!




MqttCogs Settings and Configuration

Here are all of the settings that are currently available for mqttcogs.

Most of these are directly related to connecting to your mqtt broker. There are couple of other settings for managing and archiving your data.

MQTT Version

MQTT protocol version. Pick from either 3.1 or 3.1.1. Nothing else is supported.

MQTT Server/Port

This needs to be in one of the following forms.

Either tcp://yourmqttbroker:yourportnumber OR ssl://yourmqttbroker:yourportnumber

Make sure that outgoing tcp connections to yourmqttbroker:yourportnumber are allowed by your hosting provider.

This tripped me up.

MQTT ClientID

Unique identifier. Should be unique for MQTTCogs

MQTT User

Obvious, make sure that this user password combination does actually have access to your mqtt broker and appropriate topics.

MQTT Password

Obvious, make sure that this user password combination does actually have access to your mqtt broker and appropriate topics.

MQTT TopicFilter

Topic filter applied when subscribing to messages. Set to # if you want to get all messages from your broker.

MQTT Read Role

WordPress roles that are allowed to see MqttCogs shortcode data. For example, if you set this to ‘Subscriber’ a user will need to be logged in to be able to see any data.

MQTT Write Role

WordPress roles that are allowed to send Mqtt data via a MqttCogs shortcode. For example, if you set this to ‘Subscriber’ a user will need to be logged in to be able to use your widgets. See Publish Data Using An MqttCogs Shortcode

MQTT Connection Recycle (secs)

How often MqttCogs drops the connection to the broker and reconnects. By default MqttCogs runs a ‘watchdog’ routine every minute. If it detects the connection to the Mqtt broker has dropped it reconnects.

It will also reconnect every MQTT Connection Recycle seconds. I have mine set to 298 seconds as I know that my hosting provider has a timeout of 300 seconds on my php scripts.

Save MQTT data for

MqttCogs prunes saved Mqtt data daily. You can choose how much you want to keep.

MQTT Debug

Set the debugging level. Debug information is written to php_errors.log. Normally found in the root of your site.

Avoid leaving this set to ‘All’ as this will produce very large logs! This is useful short term if you want to diagnose a problem.

MySensors Receive Topic

Only set this if you are using MySensors. This allows support for buffering outgoing messages to nodes that use the MySensors smartSleep function.

If you are receiving data from a MySensors node provide the topic that the node uses when it sends data.

For example, my nodes send to topics like this:

mysensors_out/92/2/1/0/0

I would set this to mysensors_out

MySensors Transmit Topic

Only set this if you are using MySensors. This allows support for buffering outgoing messages to nodes that use the MySensors smartSleep function.

If you are sending data to a MySensors node, provide the topic filter for this.

For example, I can send commands to my nodes in this format

mysensors_in/10/6/1/0/2

I would set this to mysensors_in




MySensors smartSleep

How to support the MySensors smartSleep function?

FYI my development board looks like this. Parts include Arduino Pro Mini, MCP1703 (+v.e to 3.3v), 8 channel relay board, nRF24L01+ radio module.

I want to be able to reliably send commands to this MySensors node but I want the node to be sleeping most of the time when the relays are not activated.

When you code up a MySensors node you can save power by sleeping. This works fine until you want to send the node a command. The node won’t receive the command because it is asleep. I initially thought that I could get around this by simply sending the command with a qos status of 1.  This doesn’t work!

Recall that a MySensors NODE communicates with a MySensors GATEWAY. The gateway is responsible for sending and receiving messages to an MQTT BROKER.

If you send a command with a qos status of 1 this guarantees delivery to the GATEWAY but it does not guarantee delivery to the node if the node is sleeping. The gateway isn’t clever enough to buffer messages for nodes.

The MySensors library has a built in function called smartSleep. When you call this function the following happens:

  1. The Node sends a message (I_PRE_SLEEP_NOTIFICATION) to the server with a payload containing the number of milliseconds before it sleeps
  2. The Node waits for this amount of time
  3. The Node sleeps

Sending a command to a sleeping node then becomes a lot easier.

The following algorithm can be used when a command is received.

  • MqttCogs receives a set command
  • Check whether node is sleeping
  • If sleeping buffer message to new table
  • If not sleeping or it doesn’t look like a MySensor packet then issue the set command

The following algorithm can be used when a incoming I_PRE_SLEEP_NOTIFICATION message is received by MqttCogs

  • Check buffer for messages for this node (extract in datetime order)
  • Send messages to Node

I’ve had an alpha stab at implementing this. The following will submit a set command to node 10, sensor 6. This is a relay, the value is the number of minutes the relay should open.

[ mqttcogs_set topic='mysensors_in/10/6/1/0/2' text='Add' label_text='Enter a new value here:' input_type='number' input_min='0' input_max='6' input_title='Please enter a number between 0 and 6' qos='1'/ ]

And heres a history of what happened. When you set the value the graph won’t change. The node is only not sleeping for 500ms every minute unless the relay is activated….

For reference, my arduino loop looks something like the code below. I added in a couple of useful safety features.

  • A relay can only be activated for a fixed time only.
  • When the arduino restarts the relays are set to OFF
void loop()
 {
 
 if (sensor_active) {
     //check if sensor should still be on
     if ((millis() - sensor_start)> sensor_ms) {
        digitalWrite(sensor_active-1 + RELAY_PIN, RELAY_OFF);
        saveState(sensor_active, false);
        send(msg.setSensor(sensor_active).set(false));
        sensor_active = 0;
        sensor_start = 0;
        sensor_ms = 0;
     }
     //don't sleep while relay is on
     return;
   }
   smartSleep(60000);
 }

So, power saving?

It’s good. While sleeping the board (without modifications) consumes 1.5mA (mainly due to the led). I removed the on LED and the board now consumes 60uA when sleeping (should be less so need to look at why).

When relay is on it uses *a lot* because the relay has to be energised.

I’ve coded things so that only one relay can be on at once. This reduces the suck through the MCP1703. The MCP1703 only supports 250mA current draw. More than 2 relays on at once would break it!




Connect Serial Flash to Arduino

How to Connect Serial Flash to Arduino to Save Power

I chose a SPI Flash or Serial Flash module for 2 reasons. Firstly, it uses a lot less power than an SD card and secondly it has a much smaller form factor. An SPI Flash module will use <10mA for writing whereas an SD card will use up to 150mA.

So my thought was ‘I can drive the Serial Flash module from one of the Arduino digital IO pins’.

I bought one of these https://www.ebay.co.uk/itm/W25Q-Winbond-Serial-Flash-Memory-Module-SPI-W25Q128B-BIOS-25Q64BVSIG-EEPROM-/181873964697. This contains a winbond 128Mbit Serial Flash chip.

The board includes pull up resistors (more about these later) and labelled pins. The board is 3.3v volts. I did all my testing for this post using a Arduino Mini Pro at 3.3v so I didn’t have to do any logic level conversion.

Why 128Mbits?

One of my previous posts included code to write a serial buffer to EEPROM. This allowed chunks of data up to 255bytes to be written sequentially to the EEPROM. On startup the code determined the next ‘free page’ to write to and only overwrote one 4K block once the log started to loop.

So 128Mbits = 16777216 bytes

So 16777216 bytes = 65536 lots of 255byte records

So if I write a single log record every minute that is 45 days of logging

Wiring the Winbond W25Q128B module to the Arduino

[table caption=”Wiring the SPI Flash Module” width=” colalign=”left|left|center|left|right”]
Arduino Pin, SPIFlash Pin
Pin 6 (your choice), VCC
GND, GND
VCC, WP/IO2
VCC, HOLD/IO3
MISO, DO/IO1
MOSI, DI/IO0
SCK, CLK
Pin 10 (SS), CS
[/table]

Enable and Disable

I firstly tried to the SPIMemory library for sleeping. There are functions for powerUp() and powerDown(). I rejected these. A powerDown and Arduino reset causes the SPI Flash module to lock up and the Arduino wouldn’t restart.

This took a bit of trial and error but I eventually came up with the following function. I’ve attached VCC of the Serial Flash to Pin 6 of

void flashToggle(bool yes) {
  if (yes) {
      pinMode(6, OUTPUT); 
      digitalWrite(6, HIGH); 
      flash.begin();
  }
  else {
      pinMode(6, INPUT); 
     digitalWrite(6, LOW); 
  }
}

Switching the pin to INPUT and setting low seemed to stop all power from the pin. I tried all other options.

Power  Use

These were measured on the VCC line with a cheap multimeter.

Continuous write ~ 5mA
after flashToggle(false) – 0uA (err, yes really. Nothing)

Conclusion

For me, a result. The module uses a tiny about of power and I can switch it off by simply toggling the DIO pin. I need to see if the same is true when it is attached to a 5v Arduino through a logic level converted.




Arduino Circular Buffer to Serial Flash

Writing an Arduino Circular Buffer to Serial Flash.

Data is arranged in BLOCKS of 4K, each block contains PAGES of length 256 bytes. Pages are written sequentially to flash.

When we get to the end of the flash we then start at the beginning. Serial Flash must be erased a BLOCK at a time. We erase only one BLOCK ahead of where we are currently writing.

On startup we determine what is the next free page in the flash so that we can continue writing. We keep an address pointer in memory after this.

The first byte of each page is used to flag teh following:

11111111 – not written

01111111 – writing

00111111 – written

The following 255 bytes contain the page payload. We CANNOT write more than 255 bytes to a page.

#define BLOCK_SIZE 4096  //4K
#define PAGE_SIZE 256    //

#define PAGE_WRITING 0x7F //01111111
#define PAGE_WRITTEN 0x3F //00111111

SPIFlash flash;
uint32_t addr;
uint32_t capacity;

void setup() {
  Serial.begin(BAUD_RATE);
   flash.begin();
   capacity = flash.getCapacity(); 
   addr = getNextAddress();
}


/**
  Writes a variable length packet of no more than 255
  bytes to the rolling log

  @param data_buffer[] the bytes to write
  @param bufferSize the size of data_buffer
  @return the next available address
*/
uint32_t writePacket(byte data_buffer[], int bufferSize) {
  if (addr%BLOCK_SIZE==0) {
     Serial.print("ERASE:");
     Serial.println(addr);
    flash.eraseSector( (addr + BLOCK_SIZE)%capacity);
  }
  flash.writeByte(addr, 0x7F); //flag as started
  flash.writeByteArray(addr+1, data_buffer, bufferSize);
  flash.writeByte(addr, 0x3F); //flag as written
  return (addr + PAGE_SIZE) % capacity;  
}


/**
  Examines the SerialFlash finds the next empty page. Should
  be called on startup. A reference can then be kept to
  log position.

  @return the next available address
*/
uint32_t getNextAddress() {
    uint32_t ad = search(0,capacity, BLOCK_SIZE, 0x3F);
    return (search(ad, BLOCK_SIZE, PAGE_SIZE, 0x3F)+ PAGE_SIZE)%capacity;
}

/**
  Examines the first byte in a chunk of flash to determine if 
  page has been written.

  @param from the from address
  @param to the to address
  @param ss the step size, normally BLOCK or PAGE
  @param headermask mask to apply to first byte of PAGE
  @return the next available address
*/
uint32_t search(uint32_t from,uint32_t to, uint32_t  ss, byte headermask) {
  byte lb = 255;//isnt written
  uint32_t pt;
  
  for ( pt = from; pt<=to;pt=pt+ss) {
    byte b = flash.readByte(pt);   
    if ((b==255) && (~lb & ~headermask)) {
      return pt-ss;
    }
    lb = b;
  }
  //got to the end
  return pt;
}



Test

This is a test page for me, for my moisture sensors. So won’t make much sense at the moment.

Problems:

  • Moisture reading fluctuation with temperature
  • Morning high moisture content – due to condensation between plate and insulator?
  • Calibration of sensors

Setup:

  • 2M resistor here (Air~83 Wet~122 @20C (130 @50C), Soil~119 @ 25.5)
  • Probe covered with epoxy resin. Try to remove issues with moisture condensing between covering and blade

Sensor 93

Sensor 92




Making a Gauge Chart

This post explains how to create a Google Visualization Gauge chart.

Extracting Data for a Google Visualization Gauge

Google Visualization Gauges need a bit of thought. If you look at the documentation for the Gauge chart here, there is a sneaky little comment that reads:

At the moment there’s no way to specify the title of a gauge chart as you can with other Google Charts

That makes things difficult. We’ll do our best to create a gauge but it won’t have a title. I’ve instead used a plugin to create 2 columns in my WordPress page and added a title above the Gauge. It’s the best I can do!

Here’s the shortcode for one of the Gauges. I simply get the top 1 row for my one of my probes.

[ mqttcogs_drawgoogle options="{height:200}"
	charttype="Gauge" 
	ajax="true" ]
		[ mqttcogs_data limit="1" topics="mysensors_out/92/1/1/0/0" ]
[ /mqttcogs_drawgoogle ]

Soil Temperature

Air Temperature

There’s obviously a bit of work for me to do here. There is no way currently of getting the maximum/min/average value for today!




Visualizing Mqtt JSON Payload 1

This post describes how to visualize Mqtt data when the Mqtt payload is JSON. I updated this post on 26th May 2019 for version 2.2. The example in this post will not work for versions prior to 2.2!

How is MqttCogs data stored?

Recall that data arriving from the Mqtt broker is persisted to a WordPress table. The data from my MySensors projects looks something like this. You can see that my data is pretty simple – numeric temperature and moisture levels.

By default, when I visualize this data I see a graph that plots utc against payload. Here is the graph that plucks out temperature data for probe 95. I’ve removed all the axes formatting to make the shortcode more readable. (Underneath the surface Mqttcogs returns a datetime and a number/string for each row of data for the Google Visualization)

[ mqttcogs_drawgoogle options="{height:300}" ajax="true" ] 
	[ mqttcogs_data from="-3" limit="99999" topics="mysensors_out/95/1/1/0/0"] 
[ /mqttcogs_drawgoogle ]

So now lets examine some data that has arrived from the Mqtt broker with a JSON payload. I’ve manually added a couple of rows. As you can see they contain latitude and longitude – can you see where this is going?

To extract and plot the values from my json payload I need to instruct MqttCogs how these should be extracted from the JSON data. Prior to version 2.2 this was done server-side. This wasn’t very flexible and only worked with very simple ‘flat’ JSON.

Version 2.2 provides the ability to manipulate the JSON in the client browser using a script. In this simple example we don’t even need to do this!

The default JSON script that is bundled with version 2.2 looks for a longitude and latitude in the JSON data (it’ll scan for lon, longitude, lng and lat, latitude) and uses that to plot the data. It also constructs a simple flag using the the topic name and datetime.

[ mqttcogs_drawgoogle charttype="Map" options="{height:300}"  ]
  [ mqttcogs_data 
	limit="99999" 
	topics="jsontest/a" ] 
[ /mqttcogs_drawgoogle ]

This great! I can now plot lat, lng on a google map. I could even get the map to update if something was moving!

The above is a really simple example introducing JSON payloads. Take a look at Part 2. This shows how we can use the ‘script’ attribute to properly manipulate the JSON data before we use it in our visualization.