When a Mongoose met a MicroPython, part II

• Tutorials
• Communications
• C Programming

Mongooses and very small pythons develop their established friendship.

In the first part of this blog, we introduced this little framework to integrate MicroPython and Cesanta's Mongoose; where Mongoose runs when called by MicroPython and is able to run Python functions as callbacks for the events you decide in your event handler. Now we add MQTT to the equation, so we can subscribe to topics and publish messages right from MicroPython. The code is completely written in C, except for the example Python callback functions, of course. To try it, you can just build this example on a Linux machine, and, with just a small tweak, you can also run it on any ESP32 board.

How to build + See it in action

Building for Linux does not have any special requirements beyond a C compiler and make, it will clone the MicroPython repository and build the code on-the-fly. If you then want to build for an ESP32, that requires Docker, as we've seen on a previous post: Bellegram.

Clone this Github repository and change to the "pythongoose" directory, then call make:

$ git clone https://github.com/scaprile/mongoose_apps.git
$ cd pythongoose
$ make

This also starts the Python script, so you should see no prompt back.

LINK build-standard/micropython
   text       data        bss        dec        hex    filename
 678684       2536       7304     688524      a818c    build-standard/micropython
make[1]: Leaving directory '/home/scaprile/work/ldir/mongoose_apps/pythongoose/micropython/ports/unix'
micropython/ports/unix/build-standard/micropython main.py
on_connect: subscribing and publishing
on_publish: got this message in embrel_src --> b'Hello, myself!'

These last two lines are the Python code handling the callbacks for the connect event and the publish event. The on_connect function subscribes to the embrel_src topic and publishes a message there. Then we receive it back, and that calls the on_publish function.

The web server functionality is still intact, so you can go to part I and try it

How does it work ?

As we've seen in part I, Mongoose is an event-driven framework that decouples us from the underlying networking stack and possible operating system, calling an event handler callback whenever there is an event of interest; as long as we periodically call its event manager. We do this inside an infinite loop in Python code. Initialization works the same as we've seen in part I. No changes there.

What is new, is that we now introduce an MQTT client. As opposed to a server, a client initiates the connection and expects to perform some function. An HTTP client, for example, will expect to store/retrieve some data to/from an HTTP server and exit. An MQTT client, instead, will connect to a broker and stay connected; hence being able to subscribe to a topic, publish to some topics, and receive messages that have been published on its topic of interest (to which it has subscribed). In the event the client would disconnect from the broker, this last one can advertise this disconnection to all interested parties. For this to work, there is something called "last will message", that a client handles to the broker at connect time. This message will then be published on the specified topic and so received by all subscribed clients. It is common to also mark this message as "retained", so it will also be delivered to clients that connect after the event, this allows to have some sort of a state indication on the connection status of a particular client, or its associated data.

As we need to have an entity that will be performing some activity on its own, and we need to be able to make it do specific actions, like publishing a message, we designed this MQTT client as an object. This client initialization is done in Python code, passing the proper callback functions for the connect and publish events:

The connection to a broker is also done at initalization time, by calling the connect method. We'll get to the parameters later, let's say that we are not passing neither user and password nor last-will message.

Once we are connected to a broker, we get called to the provided callback function. When we want to publish a message, we will call the pub method. To subscribe to a topic, we'll call the sub method. Messages we publish will be received by the broker and replicated to all its subscribed clients. Conversely, when someone publishes a message in a topic to which we are subscribed, the broker will publish it to us.

These pub and sub methods above, end up calling this Mongoose code to perform their respective actions:

The connection process is a bit more involved. We thrive to keep the connection always alive, so in case it disconnects for some aethereus reason, we connect again. This is done by starting a timer that periodically checks the connection:

Notice we mimic what could easily be a Python object from a class, holding all our internal state in a structure pointed to by celf (C-side self...)

This MQTT connection is a Mongoose connection, and as such it has an event handler. Here we run a timer to decide when a connection attempt has waited for too long and needs to be abandoned, but fundamentally we get the MG_EV_MQTT_OPEN event when the connection to the broker is established, and the MG_EV_MQTT_MSG event when the broker publishes a message in a topic to which we have subscribed. In both cases, we call their respective Python callback functions

... ...

At this point, you may be wondering how on earth does MicroPython know how to call Mongoose, or even worse, how does Mongoose know how to call a MicroPython function and how can it deal with Python objects... Well, you shouldn't be doing that if you would have read the first part of this series... Anyway, that is the task of the code that turns those weirdly named Mongoose functions into a Python module; thing that follows. But, as we will only explain what is actually new, let me insist on you reading the first part now, and only then proceeding to the next topic.

The art of MicroPython module crafting, part II: classes

As seen before, what we need to do is to write C code that interacts with Micropythons's C API, in order to morph strings and integers into Python objects, and viceversa.

The new kid in town is the class. That is, basically, a structure that identifies the object type that we will return at object creation time, a set of C functions that will be called when a class method is invoked, and another structure that ties it all together in form of a class, and is exported to the Python world.

So, our object is like this, and as it is our unit of memory allocation, it will also contain the C-side part:

When an object of this class is instantiated, Python will call the constructor, the object creator:

Here, we basically alloc memory, save the objects for the Python callbacks, populate the C-side "self" with pointers to the callback functions that will call the Python callback functions (responsible for translating C arguments to Python objects), and a link to this very object. Why ? So those callback functions can properly know who to call:

So, the Python-facing side of this split object, that is, the Python-callable methods, are C functions similar to what we've seen in part I. The connect method is perhaps a bit more involved, as it deals with a lot of arguments and the possibility for some of them to be absent. The pub and sub methods are simple:

Then, we create the object dictionary and populate it with these methods, so the MicroPython interpreter can find them. This dictionary, along with the creator function pointer, is passed to a macro that creates the necessary structure to get a full-fledged class

Finally, this class is added to the dictionary that holds the objects present in this module

Yes, just an extra line.

Let's go deeper

"A dream within a dream ?"

We start this adventure by instantiating an MQTTclient object. MicroPython calls the class constructor function, what calls the "new" function in our module. Memory gets allocated, Python callback functions registered, C callback functions registered. Our next step is then to connect to the broker. So, we invoke the connect method in that object. MicroPython calls the method function, what calls the "connect" function in our module. This function translates Python objects to C strings, then calls the C counterpart that calls a Mongoose function.

Mongoose uses a Berkeley Sockets interface to interact with the underlying networking stack. It can also provide its own TCP/IP stack, but we won't use it this time. Mongoose then interacts with Linux using Berkeley Sockets.

When we call mg_mqtt_connect(), Mongoose parses the URL and obtains the IP address to connect to, using the TCP port indicated in the URL. Default port is 1883. It then opens a socket and initiates a TCP connection to that IP and port.

When the TCP connection establishes, the MQTT connection takes place. This involves an exchange of messages and possibly user authentication, depending on how the broker is setup. In this case, we don't do user authentication. Once the connection phase is completed, we are effectively connected to the broker. This is indicated by a specific MQTT control message; with the reception of this message, Mongoose fires a specific event that we catch on our event handler. There, we call the C callback function that calls the respective Python callback function.

At this Python function, we want to subscribe to a topic and also publish a message there. Not necessarily we need to subscribe to a topic, nor publish a message, nor do both need to be the same. This actually depends on the application, what we want to do, and how we architect our application over this publish/subscribe systems architecture. The point in this example is to show/test that this is working, so we merely subscribe to a topic in which we'll publish a message, so we can receive it published back. A "topic" is a name, an indicator of a type of messages for some function that we have architected for our application. For example, a temperature sensor will be interested in publishing on a temperature topic, while a thermostat will be interested in subscribing to receive information for that topic.

So, from this "on connect" Python callback function, we first invoke the sub method, passing the desired topic and Quality of Service. The same series of actions that we've described will take place again, finally calling Mongoose's mg_mqtt_sub(). This will result in an MQTT message exchange, TCP segment exchange, IP datagram routing, electric signaling, electromagnetic fields forming waves, waves traveling, esoteric magic, ending up in the broker recording us in a list, in order to send us whatever any client will publish in that "topic". This is application dependent; in our case, we just chose an allegoric name.

Then, we invoke the pub method, passing the desired topic, message, and Quality of Service (and retain flag). The same series of actions described above will take place once more, finally calling Mongoose's mg_mqtt_pub(). This will result in yet another MQTT message exchange. The purpose of publishing, again, is application dependent, we are solving a real-life application and so we defined what to share, when, and by who; the same way we defined who will benefit from that info, and hence subscribe to that topic. MQTT is an application of the publish/subscribe systems architecture, born as a way to decouple realms of the whole application in order to solve less complex portions of the whole, one at a time. We can think of it as a queue between two tasks (and no, the 'Q' in 'MQTT' was not originally for "queue", in fact the 'MQ' comes from the name of a line of mainstream computers; 'TT' stands for Telemetry Transport).

At this point, the broker will get the topic, search its internal list, and deliver the message to every client that has subscribed to it (and is connected... should the message have the retain flag set, it would be stored and delivered on connect to new clients, but this is not the case now). We happen to be one of those, so the broker will gently publish the message back to us.

The TCP connection is still open, so any MQTT messages can go back and forth. As soon as Mongoose parses a full MQTT "publish" message, it will call our event handler function, signaling the corresponding event. There, we call the C callback function, where we craft Python objects and call the Python callback function, passing these objects as parameters.

At this "on publish" Python callback function, the application will do whatever it needs to do with that message; in our case, we just print it on the console.

Some of the deeper level internals are similar to those explained in a previous post: Bellegram. Beyond differences in OSs, let's note that here Mongoose will run every time MicroPython lets it run, when main.py calls our poll function, as seen on part I.

Sign in