Using a different tool for each job and linking them together with MQTT - part 4

Something needs to happen on the screen when you push the buttons you can see here. Except the buttons are connected to an Arduino not the Raspberry Pi that is connected to the screen.

The sequence goes something like this:

1. Push the green button and the little trapdoor opens.
2. Put a container in the revealed space and push the button again.
3. It is drawn inside the machine.
4. The trapdoor closes behind it.
5. An 'analysis' of the contents of the container begins.

I did a little video while I was testing the mechanism. The mechanism is printed in natural translucent PLA and there are some Neopixels glued on the outside so that it can indicate the 'status' of the process. If you put in a container that has already been analysed it spits it out again while glowing red.

Going back to the start though, the Raspberry Pi needs to know you've pushed the green button. We've already established in parts 1 & 2 that the Arduino is plugged in to the Raspberry Pi over USB and that a small Python script looks for serial output from the Arduino, pushing anything it gets into an MQTT topic 'arduino/out/'. Push the green button and it sends 'openLid'.

The user interface is running as a set of web pages in Chrome browser. Unless something makes them they'll set there stoically until the user clicks on a link. What's great is that Paho have made a simple Javascript MQTT client library available and this can be used to automate interaction with these pages. The library uses Websockets to connect rather than a direct MQTT connection but we've already made sure the server supports this.

Using a different tool for each job and linking them together with MQTT - part 3

Things you have to think about when putting a Raspberry Pi inside a prop include things you don't normally have to worry about like graceful startup and shutdown. Also, getting enough power out of a battery for a Pi 3 and a 5" display is not totally inconsequential.

I'll deal with the latter first. I luckily had a 5V/3A Turnigy branded battery eliminator (BEC) kicking around in a parts drawer. These are designed for RC vehicles and nicely self-contained in a shrink-wrapped module with inline inductor to smooth things out. I paired this up with a big chunk of 12 AA NiMH cells wired series/parallel to give me ~7.2V which is about right to run the BEC. The more modern solution would be some LiPo cells but then you need to build a balance charger into the device or expose the connectors so you can charge the cells individually. I've got a bunch of stuff that runs off NiMH cells and a nice high current smart charger that automatically selects voltage/current and this also influenced my decision to go this way.

Then there's the switch. People expect things to just turn on/off at the flick or press of a switch but you can't give them easy direct access to the power or they'll switch things off when they aren't ready. The Raspberry Pi really doesn't like that. There are workarounds you can do with mostly read-only filesystems but at the end of the day the Pi really should be shut down cleanly.

Luckily I had some Polulu solid state power switches lurking from a temporarily shelved project. My house may be overflowing with stuff but sometimes it comes in useful. I wired this so the power switch was connected to the 'on only' contact. That gets everything powered up. When you flick the switch to off, the alternate contact takes a pin on the Arduino low and the Arduino initiates a shutdown sequence. You saw the code for that in part 2. Here's the Python script for the shutdown of the Pi, which is just a variant on the old chestnut about fitting a shutdown switch to the case of a Pi.

#!/usr/bin/python
import time
import os
while 1:
  time.sleep(1)
  if os.path.isfile('/tmp/reboot'):
    time.sleep(5)
    os.system("sudo reboot")
  elif os.path.isfile('/tmp/poweroff'):
    time.sleep(5)
    os.system("sudo poweroff")

Not much to say about this, it just looks for files in '/tmp/' and reboots or shuts down depending on which file it sees. Stick it in /etc/rc.local again and the job's a goodun.

Things I didn't get right are I used a latching switch for on/off and I should have fitted a way to both charge the batteries and power the device externally.

The batteries lasted a couple of hours but then it had to be charged before it could be used again. We had mains power in the end so being able to use a PSU would have been good. Connecting the smart charger with it powered up made the charger shut down, otherwise that would have been an option.

The latching switch prevented me from making the box shut down when inactive. It would try to shut down (the Pi would shut down) but if the switch was physically in the 'on' position the power would stay on until you flicked the switch. I picked this up in time to fix it but couldn't find a nice momentary switch without resorting to mail order. Which would have come too late. It's a lesson learned for next time.

Using a different tool for each job and linking them together with MQTT - part 2

Now we have a working MQTT server, it's time to start making use of it.

As I was planning to use MQTT to broker messages between Web server CGI scripts, Javascript in Chrome web browser and an Arduino the obvious way to do this is push data at the USB serial port of the Arduino.

Normally this port is used for uploading the Arduino sketch to the board and sometimes people end up filling it with debug/status messages once the sketch is running. However there's a long history of it being used as an actual way to control stuff.

So I threw together a piece of 'middleware' that shuffles data between some MQTT topics and the USB serial port.

#!/usr/bin/python
import time
import os
import mosquitto
import serial
# commands to the arduino are as follow...
# R red LEDs
# G green LEDs
# B blue LEDs
# N no LEDs
# O open/go
# P power off
arduinoBootupTime = 5
arduinoIsBooted = 0
debug = True
# Define the various MQTT callback functions as this is an event driven model
def on_connect(mosq, obj, rc):
if debug:
print("rc: "+str(rc))
return
def on_message(mosq, obj, msg):
if debug:
print(msg.topic+" "+str(msg.payload))
arduino.write(str(msg.payload))
return
def on_publish(mosq, obj, mid):
if debug:
print("mid: "+str(mid))
return
def on_subscribe(mosq, obj, mid, granted_qos):
if debug:
print("Subscribed: "+str(mid)+" "+str(granted_qos))
return
def on_log(mosq, obj, level, string):
if debug:
print(string)
return
#
time.sleep(arduinoBootupTime)
# Connect to the local MQTT server to pass stuff to/from a browser
mqttc = mosquitto.Mosquitto()
mqttc.connect("localhost", 1883, 60)
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe
# Subscribe to the topic that sends commands TO the Arduino
mqttc.subscribe("arduino/in", 0)
while 1:
if debug:
print "Trying to connect to Arduino\n"
arduino = serial.Serial('/dev/ttyUSB0',115200,timeout=1)
while arduino.isOpen():
if arduinoIsBooted == 0:
if debug:
print(arduino.name) + " connected - Giving " + str(arduinoBootupTime) + "s for it to bootstrap"
time.sleep(arduinoBootupTime)
arduinoIsBooted = 1
mqttc.loop()
arduinoOut = arduino.readline()
if len(arduinoOut) > 0:
if debug:
print "Received " + arduinoOut.rstrip() + " from the Arduino"
if arduinoOut.rstrip() == 'powerDown':
powerOffFile = open('/tmp/poweroff', 'a')
powerOffFile.close()
else:
mqttc.publish("arduino/out", arduinoOut.rstrip())
arduino.close()
time.sleep(10)

This is about as simple as things can be and mostly cribbed from example scripts. It waits a while to give the Arduino time to boot as the USB port attaching to Raspberry Pi will cause it to reset. Then it tries to connect and loops round forever passing messages back and forth.

The meat of this is in two functions, the first of which gets triggered as a callback when an MQTT message arrives...

def on_message(mosq, obj, msg):
arduino.write(str(msg.payload))
return

This simply sends the message straight to the Arduino. Coming the other way is almost an exact reverse except we're using newlines to mark the end of messages so we can read them with 'readline'. This means we need to strip them off before publishing to MQTT. Python has a handy function for this in 'rstrip'.

arduinoOut = arduino.readline()
if len(arduinoOut) > 0:
if arduinoOut.rstrip() == 'powerDown':
powerOffFile = open('/tmp/poweroff', 'a')
powerOffFile.close()
else:
mqttc.publish("arduino/out", arduinoOut.rstrip())

There's another little if/else in here to handle the shutdown sequence. I used the VERY simple method of writing a file to /tmp/ if the Raspberry Pi needs to shut down, which is controlled by the Arduino. I could have had a second script subscribed to the topic for this, but this keeps all the MQTT code in one place. Because the Pi uses memory for /tmp/ instead of saving to its SD card, this safely disappears once the machine is shut down.

That's about it, things coming out of the Arduino USB serial port end up in the topic '/arduino/out/' and stuff in the topic '/arduino/in/' get sent to the serial port. The test for the existence of the Arduino serial port '/dev/ttyUSB0' means it loops round forever trying to reconnect if the Arduino get unplugged or resets.

To make this run at startup I put it into /etc/rc.local which is another quick and dirty 'make it work' solution. It works. This is not a server it's a prop, so I'm being cavalier with scalable/secure ways of doing things.

Using a different tool for each job and linking them together with MQTT - part 1

I've recently finished a quite complicated prop that needed to have a 'user interface' and thought I'd put down my thoughts on how I built it.

At a high level what we've got is a Raspberry Pi doing the user interface using Chromium web browser in kiosk mode, an Arduino Nano doing the 'physical computing' and a Teensy microcontroller acting as a custom keyboard for interaction with the Raspberry Pi.

What I've done is in principle inefficient as I could have done it all with the Raspberry Pi. However using different technologies like this is a way to compartmentalise bits of the project and use the technology you're most comfortable with for each part.

I've already used MQTT to tie things together before so it was an obvious thing to use again. It is implemented with a very simple protocol that many things are capable of understanding, including diminutive memory-constrained microcontrollers like an Arduino or ESP8266. Also, the idea of bringing data into MQTT from a hodgepodge of sources to tie things together is not a foreign concept at all, it's pretty much designed for this.

What is MQTT?

In principle the MQTT Wiki is a good point to start but like a lot of documentation in the open source community assumes a chunk of pre-knowledge and is full of gaps. So I'm going to re-invent the wheel here and describe it again in fairly plain language...

MQTT is just a way to send and receive messages. These messages can be pretty much anything you want, text, images, sounds, any arbitrary binary data. In principle they can be as large as you like but if something listening on the other end doesn't have enough memory to receive it then it'll definitely fail and probably crash or lock up.

For MQTT to work it needs a server (called a broker) that everything connects to. The broker makes sure that messages get where they need to go. A commonly used server is Mosquitto and it's what I've used. Be aware that the version which installs by default on a Raspberry Pi (probably other Linux distributions too) is old and compiled without Websockets support. You should use the current version from the Mosquitto developers or you will be limited in which clients you can use. More on that in a bit.

Topics, subscribe, publish, LWT and QoS

There are a load of clients available, Python, Javascript, Arduino, NodeMCU/Lua etc. etc. and they all use the same terminology when you want to do something with them.

Topic

A topic is a 'channel' for messages. There can be an aribtrary number of these on a broker. Unless you do specific configuration on the broker to lock things down you can create/destroy these arbitrarily by sending to or listening for data on a topic. In my application I've got topics called 'arduino/in' and 'arduino/out' for sending to and receiving from the Arduino respectively. All topics can be 'bidirectional' ie. you send and receive on the same topic from a client, but that can complicate your code as you need to process your own messages coming back at you. Which is why I'm using topics in a 'unidirectional' manner.

The broker normally handles creating topics and tidying up afterwards automatically, again unless you want to control this.

Subscribe

When you want to receive messages from a topic, you 'subscribe' to it. Depending on the programming language you use it is likely this happens as a 'callback'. This means that when you subscribe you create a function that gets run every time a message comes in on the topic. Your code needs to be able to deal with being arbitrarily interrupted when this happens.

Publish

When you want to send a message to a topic you 'publish' it. There's very little more to be said, you publish and it appears.

LWT

The broker periodically checks to make sure any clients it has are contactable. You can optionally set a 'Last Will and Testament' when you connect the client, which publishes a message to the topic of your choice if the client is no longer contactable. This is a very simple way to check if a particular client is online. I did not use this in my application.

QoS

When you publish or subscribe to a topic you can specify the 'quality of service' on the connection. This comes as...

• QoS 0: At most once, which is unreliable and the client should receive it but this is not guaranteed
• QoS 1: At least once, which is reliable. However the client may receive duplicates
• QoS 2: Exactly once, which is reliable.

Given I was dealing with two clients running on the same device as the broker I left things as QoS 0 for my application. Some clients such as the NodeMCU one only support QoS 0.

Installing Mosquitto on a Raspberry Pi

Do not be tempted to install it from the standard Raspbian repository with 'apt-get install mosquitto'. This will work but some clients will fail with unhelpful errors, particularly the NodeMCU and Javascript ones. The Javascript client requires Websockets support and NodeMCU needs MQTT v3.1.1 both of which are missing in the standard build.

There's a handy guide to installing a newer version with Websockets support here, but in case this goes away here's the potted recipe for the current version of Raspbian (Jessie) at time of writing.

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install mosquitto

Once this is installed, have a look in the file  /etc/mosquitto/mosquitto.conf and add the following line below the default 'listener'. We don't need Websockets yet, but we will later.

listener 1883

listener 9001
protocol websockets

Then restart the service with...

 sudo service mosquitto restart

Testing Mosquitto works

This is fairly easy, but first you need to install some Mosquitto clients...

sudo apt-get install mosquitto-clients

Start one session to your server and run the following command. It'll sit there waiting for messages to come in on the topic called "test".

mosquitto_sub -t 'test'

In another session, run the following command to send a message to the same topic.

mosquitto_pub -t "test" -m "Hello world!"

You should get "Hello world!" come up in the first window. It really is that simple to send messages back and forth, the client defaults to connecting to the local machine.

There was a lot of work to achieve this, but now you've done this you could have two wirelessly connected NodeMCU microcontrollers sending messages to each other via the broker and make things happen remotely. Or as we'll see in the next part, by clicking a button on a web page you can make things move that aren't directly connected to the web server.

This is all without having to create your own protocol to do this because MQTT is widely supported. I've done that kind of thing in the past and it was significantly time consuming.