Using a different tool for each job and linking them together with MQTT - part 1

I've recently finished a quite complicated prop that needed to have a 'user interface' and thought I'd put down my thoughts on how I built it.

At a high level what we've got is a Raspberry Pi doing the user interface using Chromium web browser in kiosk mode, an Arduino Nano doing the 'physical computing' and a Teensy microcontroller acting as a custom keyboard for interaction with the Raspberry Pi.

What I've done is in principle inefficient as I could have done it all with the Raspberry Pi. However using different technologies like this is a way to compartmentalise bits of the project and use the technology you're most comfortable with for each part.

I've already used MQTT to tie things together before so it was an obvious thing to use again. It is implemented with a very simple protocol that many things are capable of understanding, including diminutive memory-constrained microcontrollers like an Arduino or ESP8266. Also, the idea of bringing data into MQTT from a hodgepodge of sources to tie things together is not a foreign concept at all, it's pretty much designed for this.

What is MQTT?

In principle the MQTT Wiki is a good point to start but like a lot of documentation in the open source community assumes a chunk of pre-knowledge and is full of gaps. So I'm going to re-invent the wheel here and describe it again in fairly plain language...

MQTT is just a way to send and receive messages. These messages can be pretty much anything you want, text, images, sounds, any arbitrary binary data. In principle they can be as large as you like but if something listening on the other end doesn't have enough memory to receive it then it'll definitely fail and probably crash or lock up.

For MQTT to work it needs a server (called a broker) that everything connects to. The broker makes sure that messages get where they need to go. A commonly used server is Mosquitto and it's what I've used. Be aware that the version which installs by default on a Raspberry Pi (probably other Linux distributions too) is old and compiled without Websockets support. You should use the current version from the Mosquitto developers or you will be limited in which clients you can use. More on that in a bit.

Topics, subscribe, publish, LWT and QoS

There are a load of clients available, Python, Javascript, Arduino, NodeMCU/Lua etc. etc. and they all use the same terminology when you want to do something with them.

Topic

A topic is a 'channel' for messages. There can be an aribtrary number of these on a broker. Unless you do specific configuration on the broker to lock things down you can create/destroy these arbitrarily by sending to or listening for data on a topic. In my application I've got topics called 'arduino/in' and 'arduino/out' for sending to and receiving from the Arduino respectively. All topics can be 'bidirectional' ie. you send and receive on the same topic from a client, but that can complicate your code as you need to process your own messages coming back at you. Which is why I'm using topics in a 'unidirectional' manner.

The broker normally handles creating topics and tidying up afterwards automatically, again unless you want to control this.

Subscribe

When you want to receive messages from a topic, you 'subscribe' to it. Depending on the programming language you use it is likely this happens as a 'callback'. This means that when you subscribe you create a function that gets run every time a message comes in on the topic. Your code needs to be able to deal with being arbitrarily interrupted when this happens.

Publish

When you want to send a message to a topic you 'publish' it. There's very little more to be said, you publish and it appears.

LWT

The broker periodically checks to make sure any clients it has are contactable. You can optionally set a 'Last Will and Testament' when you connect the client, which publishes a message to the topic of your choice if the client is no longer contactable. This is a very simple way to check if a particular client is online. I did not use this in my application.

QoS

When you publish or subscribe to a topic you can specify the 'quality of service' on the connection. This comes as...

• QoS 0: At most once, which is unreliable and the client should receive it but this is not guaranteed
• QoS 1: At least once, which is reliable. However the client may receive duplicates
• QoS 2: Exactly once, which is reliable.

Given I was dealing with two clients running on the same device as the broker I left things as QoS 0 for my application. Some clients such as the NodeMCU one only support QoS 0.

Installing Mosquitto on a Raspberry Pi

Do not be tempted to install it from the standard Raspbian repository with 'apt-get install mosquitto'. This will work but some clients will fail with unhelpful errors, particularly the NodeMCU and Javascript ones. The Javascript client requires Websockets support and NodeMCU needs MQTT v3.1.1 both of which are missing in the standard build.

There's a handy guide to installing a newer version with Websockets support here, but in case this goes away here's the potted recipe for the current version of Raspbian (Jessie) at time of writing.

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install mosquitto

Once this is installed, have a look in the file  /etc/mosquitto/mosquitto.conf and add the following line below the default 'listener'. We don't need Websockets yet, but we will later.

listener 1883

listener 9001
protocol websockets

Then restart the service with...

 sudo service mosquitto restart

Testing Mosquitto works

This is fairly easy, but first you need to install some Mosquitto clients...

sudo apt-get install mosquitto-clients

Start one session to your server and run the following command. It'll sit there waiting for messages to come in on the topic called "test".

mosquitto_sub -t 'test'

In another session, run the following command to send a message to the same topic.

mosquitto_pub -t "test" -m "Hello world!"

You should get "Hello world!" come up in the first window. It really is that simple to send messages back and forth, the client defaults to connecting to the local machine.

There was a lot of work to achieve this, but now you've done this you could have two wirelessly connected NodeMCU microcontrollers sending messages to each other via the broker and make things happen remotely. Or as we'll see in the next part, by clicking a button on a web page you can make things move that aren't directly connected to the web server.

This is all without having to create your own protocol to do this because MQTT is widely supported. I've done that kind of thing in the past and it was significantly time consuming.