Version 3.0 is finally here. Here’s a brief overview of some of the new features.
Settings
Added a new tab ‘Mqtt Data’. This shows the last mqtt received messages order by utc desc. This is really useful for debugging.
Thing Page Type
Rather than just using short codes to display mqtt data. I added in the concept of a ‘Thing Page Type’. You can now create a ‘Thing’ and set associated thing properties – in particular, the associated mqtt topic.
When you edit a thing, you create a post like any other but have the option of inserting ‘canned’ mqttcogs shortcodes into the editor really quickly.
You can also replace the ‘hard’ mqttcogs topic with the thing slug if you wish. This is pretty neat as it allows you to replace a sensor, you only need to update the mqtt topic at the post level rather than each shortcode that uses it.
The reason is that I also want to be able to associate the Thing with a geographic location or shape. So you can include a GeoJSON fragment as well. This can be used if you decide you want to use a leaflet map.
Thing Templates
I’ve included some basic templates that you can select *really quickly*. For example
Datatable with last 10 items
A leaflet map
The last 7 days as a line chart
When you press the add button the template will be used to add the shortcode to your thing page. You can then replace the topic with the page slug!
If you want you can also set up your own templates!
Have you tried using the Google Visualization Table chart? It isn’t great. Just about acceptable, if you want to dump some data but is difficult to style and is missing a lot of features that you get with other table components. In this post we’ll compare the Google Visualization Table and also the new mqttcogs_drawdatatable, this uses DataTables to draw a table.
Lets try a simple google table first. We’ll show some temperatures from one of my Minew sensors. I’ve actually added some basic css styling to this because I really dislike the standard google styling
It is possible to click on the headers and change the ordering. It is also possible to set the column headers (I think!). So for a basic table it ‘sort of’ works. One area that I really dislike is the refresh. The table flickers as it appears to be completely redrawn.
Lets compare with the new mqttcogs_drawdatatable. This uses DataTables to draw the table. Like the google visualization and leaflet visualizations this includes a standard javascript file that draws the table. You can replace this with your own version. Out of the box, it looks like this.
You get a much cleaner table and the ability to customize using the DataTables library. This is the default clientside javascript that is used to draw the table (used if you don’t replace with your own). The data parameter being a Google Datatable, the datatable parameter the DataTable instance and the datatableoptions parameter the options as specified in the shortcode.
function (data, datatable, datatableoptions) {
if (!this.datatable) {
//we have to inject html for column headers
//datetime, topic1, topic2, topic3...
var headerelement = jQuery('#' + this.id + '>thead>tr').first();
for (var cidx=0;cidx<data.getNumberOfColumns();cidx++) {
headerelement.append('<th>' + data.getColumnId(cidx) + '</th>');
}
datatable = jQuery('#' + this.id).DataTable(this.options);
this.datatable = datatable;
//datatables can't handle nulls....
for(var cidx=0;cidx<datatable.columns().length;cidx++) {
datatable.column(cidx).defaultContent = "";
}
}
datatable.clear();
for (var ridx=0;ridx<data.getNumberOfRows();ridx++) {
var row = [];
var val = null;
for (var cidx=0;cidx<data.getNumberOfColumns();cidx++) {
val = data.getValue(ridx, cidx);
if (val instanceof String) {
row.push(val);
}
else if (val instanceof Date) {
row.push(val);
}
else if (val instanceof Object) {
row.push(JSON.stringify(val));
}
else {
row.push(val);
}
}
datatable.row.add(row);
}
datatable.draw();
Version 2.3 comes with some very simple css classes. In this post we are going to use a single css class group-card.
Using this class you can now very easily create a ‘card’ layout that flows using the Gutenburg editor. The mqtt-card css must be applied to a Gutenberg container, as you can’t apply custom css to a shortcode block. In this example we’ll use a Gutenberg group block.
Step 1: Create a Group Block
Create a group block using the Gutenberg editor. I’ve done this below and added an H4 block and a simple shortcode to show a graph.
Column 1
All you need to do now is select the outer GROUP block (not any of the content blocks) and add the group-card css class in the advanced section. This is in the bottom right, you may have to expand the ‘Advanced’ accordion.
After you’ve added the appropriate css you should get something looking like this. If you want you can change the background colour. It is set to an off white colour so that you can see that you’ve applied the card css correctly.
Column 1
The above card is ‘ok’ but stretches across the whole page. Not fantastic. If you want a number of cards on one page then use a columns block and move your group into the columns block. I’ve done this here. I’ve used a 3 column layout and changed my graph type to Sparkline.
I’ve added my shortcode to each of the cards. The benefit of the column layout is that the cards will now ‘flow’ as you change the size of the page. The css is really simple. It looks like this so you can change it if required.
This is another post that describes how to manipulate the Mqtt payload. The JSON payload in this example comes from a commercial Minew BLE gateway.
You must be using Version 2.3 or greater for this example to work!
For this example, we’ll use some data from a Minew gateway. The data coming from the Minew gateway has a topic of minew/ac233fc04bba/status and looks like this. The ac233fc04bba is the gateway mac address.
So the payload from the gateway is an array of sensor objects. I want to use the data from the sensor with mac address AC233FA2495C. I’m interested in the battery, temperature and humidity values. The sensor is placed in one of my vegetable beds in full sun in front of my house. I expect it to get relatively hot (and cold).
From Version 2.3 we theoretically can use the JSON extract function to get data directly from this field.
THIS WON’T WORK – UNLESS SENSOR AC233FA2495C is always in array position 0, I can’t guarantee this.
Why, JSON_EXTRACT is limited in it’s features. It can extract values from a known JSON format but not from a format based on a criteria – as far as I can tell.
I get around this by exploding the JSON from gateway as it comes in using a hook. I want a separate database entry for each sensor.
//Called as the data comes in but before it is persisted
//to the database. Create multiple rows, one for each sensor
//from the minew gateway json
function mqttcogs_msg_in_pre_minew($publish_object)
{
$topic = $publish_object->getTopic();
//Do nothing if it isn't minew data
if (!(substr( $topic, 0, 5 ) === "minew")) {
return $publish_object;
}
//get an array of objects from the payload
$mqttmsg = $publish_object->getMessage();
$payload_arr = json_decode($mqttmsg);
if (!is_array($payload_arr)) {
$payload_arr = array($payload_arr);
}
$publish_object_arr = array();
//gateway topic is /gw/ac233fc04bba/status
//extract the gateway id from this
$topic = explode('/', $topic);
$gw = $topic[2];
//the incoming payload is an array of objects like this
// {
// "timestamp": "2020-01-11T09:54:26Z",
// "type": "S1",
// "mac": "AC233FA249C9",
// "bleName": "",
// "rssi": -41,
// "battery": 100,
// "temperature": 20.79,
// "humidity": 57.05
//}
//So we loop and create a new publish object for each one
foreach($payload_arr as $key => $payload) {
//type filter...
$pb = clone $publish_object;
//for each object we want to make sure the utc field is correct
//and remove the timestamp from the json as it isn't required
if (property_exists($payload, 'timestamp')) {
$pb->setDateTime(new DateTime($payload->timestamp));
unset($payload->timestamp);
}
if (property_exists($payload, 'mac')) {
$pb->setTopic('minew/'.$payload->mac);
unset($payload->mac);
}
$payload->gw = $gw;
$pb->setMessage(json_encode($payload));
array_push($publish_object_arr, $pb);
}
return $publish_object_arr;
}
add_filter('mqttcogs_msg_in_pre', 'mqttcogs_msg_in_pre_minew', 10, 2);
After this hook is running I get a separate row for each sensor in my database. The image shows the change from the original format to the new one after applying the code above.
Not only can you visualize data using Mqttcogs but it is also possible to publish data to your Mqtt broker using the [mqttcogs_set] shortcode. The ability to publish data is useful, here are a couple of reasons I wanted to do this.
This article is based on Version 2.3 of the the MqttCogs plugin.
Number 1: I can publish data directly to my blog without having to use a generic Mqtt client.
Number 2: In the IOT (internet of things) world iot devices can be controlled by publishing appropriate messages. For example, you might be able to control your outside lights by publishing to a topic such as myhouse\lights\outside.
Remember though – you can’t have just ANYONE publishing to your broker!!
Lets start with a basic example
Here is the code for the graph and underneath the shortcode for publishing to the tests/blog/publishingdata topic. The results are shown underneath.
[ mqttcogs_drawgoogle options="{width:'100%'}" ]
[ mqttcogs_data topics="tests/blog/publishingdata" ]
[ /mqttcogs_drawgoogle]
[ mqttcogs_set topic='tests/blog/publishingdata'
minrole='Subscriber'
label_text='Enter a new value here:' ]
I have used the [ mqttcogs_set ] shortcode. You’ll notice that the field is automatically populated with the last value stored in the database. I’ve used the minimum number of attributes in the shortcode that I could get away with.
topic – the fixed topic to send the new messages to minrole – the minimum wordpress role that is required for the fields to be displayed label_text – the label shown above the field
….but you’ll notice a problem! There is no field validation. I could enter anything into the field and it will published!!
You’ll also notice that adding a new value doesn’t automatically update the graph. The graph refresh is independent of the ‘add’. There is no link between the add and the visualization. You will need to wait until the next ‘refresh’ time for the graph to update.
Lets add some validation!
Validation is applied in the browser by using HTML5 validation. There are a number of attributes starting ‘input_’. Each is mapped directly to a HTML5 attribute on the input element. You can even change the input type to ‘range’ or ‘number’ but more about this later.
Ok, so we’ll update the attributes in the shortcode to make the field accept numeric integer values only between 0 and 50.
[ mqttcogs_set
topic='tests/blog/publishingdata'
minrole='Subscriber'
label_text='Enter a new value here:'
input_type='number'
input_min='0'
input_max='50'
input_title='Please enter a whole number between 0 and 50' ]
We’ve now restricted the field to accept values between 0 and 50. It’ll also notify the user if the input is invalid. Our numerical entry now looks like this:
Validation to 2 decimal places
I might decide that actually I want to allow decimals, say to 2 decimal places. I could achieve this using the HTML5 step attibute:
[ mqttcogs_set
topic='tests/blog/publishingdata'
minrole='Subscriber'
label_text='Enter a new value here:'
input_type='number'
input_min='0'
input_max='50'
input_step='0.01'
input_title='Please enter a number between 0 and 50 to 2dp maximum' ]
You can even do sliders
You can do this by setting the input_type to ‘range’. What is rendered will depend on your browser. The result isn’t particularly pretty in firefox. There is no feedback showing what the current slider value actually is.
A full list of supported HTML5 shortcode attributes are documented here
Ok, so this isn’t very pretty?
Being pretty has never been my strongpoint. There is also a ‘class’ shortcode attribute. So we can tidy things up by specifying a class to apply
[ mqttcogs_set
topic='tests/blog/publishingdata'
text='Add'
label_text='Enter a new value here:'
minrole='Subscriber'
input_type='number'
input_min='0'
input_max='50'
input_title='Please enter a number between 0 and 50'
class='mqttcogs_set' ]
My css looks something like this. I added this under the Appearance…Custom CSS option in the wordpress menu.
Checkboxes are easy. You can simply set the input_type to ‘checkbox’.
[ mqttcogs_set
topic='tests/blog/publishingdata'
text='Add'
label_text='Enter a new value here:'
minrole='Subscriber'
input_type='number'
input_min='0'
input_max='50'
input_title='Please enter a number between 0 and 50'
]
Please log in to be able to publish to this topic
Making a toggle button is again relatively easy. We add a custom class to the shortcode for styling. I’ve called this ‘android-switch’ but it can be whatever you like.
[ mqttcogs_set
topic='tests/blog/publishingdata'
text='Add'
label_text='Enter a new value here:'
minrole='Subscriber'
input_type='number'
input_min='0'
input_max='50'
input_title='Please enter a number between 0 and 50'
class='android-switch'
]
We then add some styling to the checkbox to make it into a toggle button
We’ve already seen that it is possible to show a simple plot of longitude and latitudes from a JSON payload. See Visualize OwnTracks GPS Data.
If you like simple Google maps this works but isn’t particularly flexible. You can only display points on the Google map using the very limited Google Map Visualization. If you also want to track ‘assets’ you’ll be liable at some point to pay a fee per asset. This is not good!
I added the ability to visualize using the Leaflet mapping library. You can set this up to use open source maps and the library is flexible enough to allow fancy stuff like polylines to be drawn!
Lets draw a Leaflet map in a similar way to our Google map. This is really easy. Just use an mqttcogs_drawleaflet shortcode. This draws points in a similar way to a Google Map Visualization in the link above.
Version 2.2 includes the ability to group and aggregrate data. This is really useful, it allows you slice data by day and apply operations like MIN, MAX and AVG.
We’re going to work through an example that will show the MINimum temperature and the MAXimum temperature for each day for one of my sensors.
The new attributes are applied to the mqttcogs_data shortcode. The resulting grouping and aggregation is done at the server*.
We’ll dive straight into the example to see how they work.
Here’s the corresponding explanation of each of the attributes for this example.
topics: Extracts temperature data for the topic minew/AC233FA2495C$.temperature. It extracts data for the same topic twice as we want to compare max and min values.
from: Restricts data to the last 30 days
group: Groups the data for each topic based on the DAY. Note that grouping can be anything that the MySQL EXTRACT funtion supports.
aggregrations: Applies the aggregation to the grouped data for each topic. This attribute should contain a comma delimited list of aggregations. One for each topic. Note that the aggregation can be anything that the MySQL group by operation supports.
This post describes how to manipulate the Mqtt payload when it contains JSON data. You must be using Version 2.2 or greater for this example to work!
For this example, we’ll use some owntracks data. See Visualizing Owntracks Data for an introduction. Firstly, lets remind ourselves of what the data looks like in our database.
My requirements are simple. I want to write a script that will plot my location data on a Google Visualization Map and include a flag that shows the Latitude, Longitude and fix time.
If we examine the documentation for the Map visualization we can see that we need to create data containing fields lat, lng and description. The google map visualization then takes care of the rest.
We are going to use the mqttcogs ‘script’ attribute. This attribute allows a user to apply a javascript function to the data client side. This allows a developer to make full use of the Google Visualization datatable and dataview javascript classes. You can manipulate, filter, group, create calculated columns and format the data before the chart is drawn!
Before we look at a function remember that the script attribute points to a wordpress custom field. The custom field contains the script. This is necessary as we can’t embed a javascript function directly into the shortcode.
The custom field section is located at the bottom of a post. You will need to make it visible via the Gutenberg editor options.
Now we are ready to examine a typical script. In fact, we’ll look at the default script that is applied if you don’t provide one yourself. Remember this is a javascript function. The parameters include
data – a google visualization datatable containing the mqtt data
chart – a google visualization chart, created using the charttype attribute you supplied
options – a javascript object created from the options attribute you supplied in the shortcode
The above code is very simple. We check if the chart is a Map, if it isn’t we simply draw the chart using the datatable.
If the chart is a Map, then we create a DataView with three columns. Columns 1 and 2 are latitude and longitude (we do our best to find them as they could be named differently depending on the data source) and Column 3 is a string composed of the topic and the utc (datetime). The visualization is then drawn using the view rather than the datatable.
This is useful for a simple data dump of Mqtt data. I’ve tried to use this visualization in other projects but I found that the table is a little difficult to format and doesn’t include features that other table libraries do a lot better. See DataTables, for example!
Firstly we’ll have a little look at some Mqtt data from the database. You can dump the JSON data directly to the browser by using an mqttcogs_data shortcode on it’s own.
In a Create a Time Triggered Action we were able to schedule an Mqtt Message (the action) to be sent at a given time or interval. This example shows how to take an action based on an incoming message. I’ve highlighted my criteria in green and my action in red
When I receive a moisture low message AND it is after 6pm AND watering is off I want to start watering my plants.
Incoming Message (from sensor)
Incoming messages from my sensor have a topic of mysensors_out/92/0/1/0/37. The payload contains the moisture level, anything above 100 is ok, anything below 100 is ‘dry’.
Outgoing Message (to sensor)
In order to turn on my pump I send a message to the mysensors_in/10/6/1/0/2 topic. The payload contains the number of minutes that the pump should be on.
So lets have a first go at implementing our logic. This won’t include ‘debounce’ and detecting whether watering is off. We’ll get this one working first and then look at implementing the other bits. We’ll use our standard after_mqttcogs_msg_in Action Hook.
add_action('after_mqttcogs_msg_in', 'moisture_msg_in', 10, 3);
function moisture_msg_in($utc, $topic, $payload)
{
//this is not the sensor you are looking for, move along
if ($topic != 'mysensors_out/92/0/1/0/37') {
return;
}
//it is not dry, move along
$payload = (int) $payload;
if ($payload>=100) {
return;
}
//it is not after 6pm, move along
$dtm = new DateTime();
$dtm->setTimezone(new DateTimeZone('Europe/London'));
if ((int)$dtm->format('H'))<18) {
return;
};
//it is after 6pm, it's dry so turn the water on!
$outtopic = 'mysensors_in/10/6/1/0/2';
$outpayload = 1;
$mqttcogs = new MqttCogs_Plugin();
$mqttcogs->sendMqtt('moisture_msg_in', $outtopic, $outpayload, 1, 0, '');
}
All pretty easy so far. We are left with one final criteria to check. We need to decide whether the watering is currently ‘on’ or ‘off’.
MqttCogs provides some neat function calls so that you can easily query the Mqtt data stored in the database. For this case, we’ll simply find the last mysensors_out/10/6/1/0/2 message from the device and look at it’s payload. A payload set to ‘0’ means that watering is ‘off’. A positive payload means watering is on.
//watering is currently active, move along
$rows = $this->getLastN('data', 'mysensors_out/10/6/1/0/2', 1, 'DESC');
//check there is a last record and if it's payload is not
//zero then we are watering
if ((count($rows)==1) && ($rows['payload'] > 0)) {
return;
}
So here is the final version. This includes a check for all the criteria at the top of the post and some simple error handling to email me when watering is on/buffered or we fail for some reason.
add_action('after_mqttcogs_msg_in', 'moisture_msg_in', 10, 3);
function moisture_msg_in($utc, $topic, $payload)
{
$debug = false;
//this is not the sensor you are looking for, move along
if ($topic != 'mysensors_out/92/0/1/0/37') {
return;
}
//it is not dry, move along
$payload = (int) $payload;
if ((!$debug) && ($payload>=100)) {
return;
}
//it is not after 6pm local time, move along
$dtm = new DateTime();
$dtm->setTimezone(new DateTimeZone('Europe/London'));
if ((!$debug) && ((int)$dtm->format('H'))<18) {
return;
};
//watering is currently active, move along
$rows = $this->getLastN('data', 'mysensors_out/10/6/1/0/2', 1, 'DESC');
if ((count($rows)==1) && ($rows['payload'] > 0)) {
return;
}
//it is after 6pm, it's dry and watering is off - so turn the water on!
$outtopic = 'mysensors_in/10/6/1/0/2';
$outpayload = 1;
$mqttcogs = new MqttCogs_Plugin();
$error = new stdClass();
//note that this function returns
// true (message ok or buffered) or
// false for error
// I just use the $error object though...
$mqttcogs->sendMqtt('moisture_msg_in', $outtopic, $outpayload, 1, 0, $error);
switch($error->status) {
case 'ok':
wp_mail( 'test@test.com', 'Watering on (sent)', 'Moisture is: '.$outpayload );
break;
case 'buffered':
wp_mail( 'test@test.com', 'Watering on (buffered)', 'Moisture is: '.$outpayload );
break;
case 'error':
wp_mail( 'test@test.com', 'Watering FAIL', $error->errors[0].reason);
break;
}
}
