Giter Club home page Giter Club logo

emqtt5's Introduction

eMQTT5

An embedded MQTTv5 client in C++ with minimal footprint, maximal performance.

X-Ryl669

This repository contains a complete MQTT v5.0 client that's optimized for code size without sacrifying performance. This is, to my knowledge, the smallest (and complete!) MQTT v5.0 client for embedded system with a binary size down to less than 17kB on ESP32 (and less than 75kB on MacOSX). MQTT v5.0 is a more complex protocol than MQTT v3.1.1, with the addition of properties in each packet and authentication subsystem.

Why another MQTT client ?

For many reasons:

  • Many clients around don't support MQTT v5.0 protocol (only limited to version 3.1.1)
  • Some are large and/or requires numerous dependencies
  • This code is specialized for embedded system with or without an operating system
  • Many clients don't build on a linux system making debugging hard
  • The license to use them is too restrictive
  • Some client rely on a heap and fragment the heap quickly making usage over a long period dangerous

Comparison with existings clients I know about

Client Supported MQTT version License Compiled code size (with dependencies) Cross platform
256dpi esp-mqtt 3.1 MIT 11kB (113kB + ?) No (ESP32)
Espressif esp-mqtt 3.1 Apache 2.0 12kB (115kb + ?) No (ESP32)
wolfMQTT 5.0 GPL 2.0 not tested due to license Yes (Posix+Win32+Arduino)
mosquitto 5.0 EPL large Yes requires Posix
eMQTT5 5.0 MIT <17kB (no dep) Yes (Posix+Win32+Lwip(for ex: ESP32))

API Documentation

You'll find the client API documentation here.

There are two levels to access this client. The low level implies dealing with packet construction, serialization (without any network code). It's documented here.

The higher level API which is documented here is available when you only need to call methods of the Network::Client::MQTTv5 class (all serialization is done for you).

In all cases, almost all methods avoid allocating memory on the heap (stack is prefered whenever possible). There is only few places where heap allocations are performed and they are documented in there respective documentation.

Typically, user-generated Properties are allocating on the heap (in your code, not here) and user-generated SubscribeTopic are also allocating on the heap (in your code, not here).

An example software is provided that's implementing a complete MQTTv5 client in MQTTc.cpp where you can subscribe/publish to a topic. This file, once built on a Linux AMD64 system takes 80kB of binary space without any dependencies.

Porting to a new platform

The implementation for a new platform is very quick.

The only dependencies for this client rely on a Lock class to protect again multithreading access/reentrancy and a ScopedLock RAII class for acquiring and releasing the lock upon scope leaving. A default spinlock class is provided in the minimal implementation.

BSD socket API is used with only minimum feature set (only recv, send, getaddrinfo, select, socket, close/closesocket, setsockopt is required).

The only options used for socket (optional, can be disabled) are: TCP_NODELAY, fcntl/O_NONBLOCK, SO_SNDTIMEO, SO_RCVTIMEO)

Please check the MQTTClient.cpp file for two different examples of platform support (from complete, deterministic, Posix based implementation to simplest embedded system).

There is also a port for ESP32 here.

MQTTv5 Packet parser

In addition to the client, the tests folder contains a MQTT packet parser for MQTT v5.0. It's built by default and used like this:

$ # Give it the raw bytes from network communication and it'll dump what it means
$ ./MQTTParsePacket 30 1E 00 18 73 74 61 74 75 73 2F 59 4F 4C 54 79 79 76 75 57 58 50 5A 2F 6C 6F 67 73 00 5B 31 5D
Detected PUBLISH packet
with size: 32
PUBLISH control packet (rlength: 30)
  Header: (type PUBLISH, retain 0, QoS 0, dup 0)
  PUBLISH packet (id 0x0000): Str (24 bytes): status/YOLTyyvuWXPZ/logs
  Properties with length VBInt: 0
  Payload (length: 3)

You can also give it a file containing the capture of the network payload:

$ ./MQTTParsePacket -f capture.dump
Detected PUBLISH packet
with size: 32
PUBLISH control packet (rlength: 30)
  Header: (type PUBLISH, retain 0, QoS 0, dup 0)
  PUBLISH packet (id 0x0000): Str (24 bytes): status/YOLTyyvuWXPZ/logs
  Properties with length VBInt: 0
  Payload (length: 3)

emqtt5's People

Contributors

x-ryl669 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

emqtt5's Issues

Linux specific int types

in LinuxSpecific.cpp [0] I have to change the typedef, otherwise it will not compile:

    typedef uint32_t __u32;         /* hack to avoid including kernel header in userspace */
    typedef uint16_t __u16;         /* ditto */
    typedef uint8_t __u8;           /* ditto */
    // For camir version of #include <linux/ethtool.h>
    typedef uint32_t  u32;            /* hack to avoid including kernel header in userspace */
    typedef uint16_t  u16;            /* ditto */
    typedef uint8_t   u8;             /* ditto */
    typedef uint64_t  u64;            /* ditto */

[0]

typedef __uint32_t __u32; /* hack to avoid including kernel header in userspace */

Clean source dependency for readability

I found some improvement for source readability during #4 .

#include <something/something.hpp>

is better than:

#include "../../something/something.hpp"

because ../../ is too strong method we can indicate any files without path setting.

There seems no definition of Platform::free and so on for macOS.

my clang says "Undefined symbols". Does eMQTT5 support macOS?

Undefined symbols for architecture x86_64:
  "Platform::free(void*, bool)", referenced from:
      Network::Client::MQTTv5::connectTo(char const*, unsigned short, bool, unsigned short, bool, char const*, Protocol::MQTT::Common::DynamicBinDataView const*, Protocol::MQTT::V5::WillMessage*, Protocol::MQTT::V5::QualityOfServiceDelivery, bool, Protocol::MQTT::V5::Properties*) in libeMQTT5.a(MQTTClient.cpp.o)
      Network::Client::MQTTv5::Impl::Impl(char const*, Network::Client::MessageReceived*, Protocol::MQTT::Common::DynamicBinDataView const*) in libeMQTT5.a(MQTTClient.cpp.o)
      Protocol::MQTT::Common::DynamicString::readFrom(unsigned char const*, unsigned int) in libeMQTT5.a(MQTTClient.cpp.o)
      Network::Client::MQTTv5::Impl::~Impl() in libeMQTT5.a(MQTTClient.cpp.o)
  "Platform::malloc(unsigned long, bool)", referenced from:
      Network::Client::MQTTv5::connectTo(char const*, unsigned short, bool, unsigned short, bool, char const*, Protocol::MQTT::Common::DynamicBinDataView const*, Protocol::MQTT::V5::WillMessage*, Protocol::MQTT::V5::QualityOfServiceDelivery, bool, Protocol::MQTT::V5::Properties*) in libeMQTT5.a(MQTTClient.cpp.o)
      Network::Client::MQTTv5::Impl::Impl(char const*, Network::Client::MessageReceived*, Protocol::MQTT::Common::DynamicBinDataView const*) in libeMQTT5.a(MQTTClient.cpp.o)
  "Platform::realloc(void*, unsigned long)", referenced from:
      Protocol::MQTT::Common::DynamicString::readFrom(unsigned char const*, unsigned int) in libeMQTT5.a(MQTTClient.cpp.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Porting to Renesas MCU without OS

Hi,

I tried to port it to RA6M5 Renesas MCU. In Types.hpp, there is several pre-processors macro such as _WIN32, _POSIX, and ESP_PLATFORM. As RA6M5 is not fall under one of them, I'm not defining anything. When I tried to compile it, there is requirement to include pthreadRTOS.h. Even after I make empty header file just to pass compilation, at the end of the day it still requires me to define pthread_t.

On README.md, it said that This code is specialized for embedded system with or **without** an operating system. AFAIK, to add thread into embedded system, we need to put Real Time OS (FreeRTOS, Azure RTOS, etc). So, my question is, how can I compile it without RTOS dependencies? Is there any pre-processor that I need to define? Or did I misinterpret something? Thank you for your assistance.

Problems including the library

Hello,
I am having trouble including the library and I am not sure if I installed or used it correctly.
Sorry, I am not an experienced programmer. I tried to google how cmake should be used, but I haven't found a solution.

I use Ubuntu 20.04 and tried this:

  • cloned the project to ~/projects/eMQTT5

  • used those commands:
    cmake CMakeLists.txt
    make
    sudo make install
    sudo ldconfig

  • found MQTTc in /usr/local/bin and libeMQTT5.a in usr/local/lib

  • but there is nothing in /usr/local/include

  • in my project which is located in ~/projects/device/src I used
    #include "../../eMQTT5/lib/include/Network/Clients/MQTT.hpp"
    (#include "Network/Clients/MQTT.hpp" did not work)
    but I get include errors in MQTT.hpp :
    #include <Network/Clients/MQTTConfig.hpp> and
    #include <Protocol/MQTT/MQTT.hpp> can't be found

Thanks for your help!

Publisher and Subscriber in the same client

I want to implement publisher and subscriber in the same client. Publisher will send data in every 1 second and subscriber will listen for any incoming message in another topic. Can you provide me with any example?

readFrom() return BadData for 4-byte buffer

In [0] there is the check of the variable byte integer buffer length

uint32 readFrom(const uint8 * buffer, uint32 bufLength)
{
    for (size = 0; size < 4;)
    {
        if ((uint32)(size+1) > bufLength) return NotEnoughData;
        value[size] = buffer[size];
        if (value[size++] < 0x80) break;
    }
    return size < 4 ? size : (uint32)BadData;
}

for buffer = {0x8c, 0xda, 0xc4, 0x09} and bufLength = 4 this will result in BadData but it should be a valid input.
Is the return condition correct?

[0] https://github.com/X-Ryl669/eMQTT5/blob/2f261615d3c2adc5e0c273876cfb35a0c7304b95/lib/include/Protocol/MQTT/MQTT.hpp#L692C31-L692C31

Adding properties to PUBLISH packet.

Hi! I tried to pass some props to "publish" method.

...

Protocol::MQTT::V5::registerAllProperties();
Network::Client::MQTTv5::Properties properties;
if (!correlation_data.empty()) {
       Network::Client::MQTTv5::DynamicBinDataView c_data(correlation_data.length(), (const uint8_t*)correlation_data.c_str());
       Protocol::MQTT::V5::Property<Network::Client::MQTTv5::DynamicBinDataView> prop(Protocol::MQTT::V5::CorrelationData, c_data);
       if (!properties.append(&prop))
           printf("PROP APPEND FAILED!\n");
       else
           printf("PROP APPEND SUCCESS!\n");
}
client.publish(publish_topic, (const uint8*) publish_message,
                                           strlen(publish_message), retain_published_message, QoS, (uint16)0U, &properties))

But, i recieve

pure virtual method called
terminate called without an active exception

in

uint32 copyInto(uint8 * buffer) 
{
    o = 1; buffer[0] = header.typeAndFlags;
    o += remLength.copyInto(buffer+o);
    o += fixedVariableHeader.copyInto(buffer+o);
    o += props.copyInto(buffer+o);  //HERE
    o += payload.copyInto(buffer+o);
    return o;
}

Maybe i'm just doing something really stupid, but i can't make it work.
But if i just create empty Network::Client::MQTTv5::Properties properties; it works (without any property ofc)

Stuck in eventLoop

I run your esp port (esp-eMQTT5) on an ESP32. Note that I'm using MbedTLS with self signed certificates to secure the connection.

Everything works great except for running the eventLoop from another thread.
The thread creation and passing the client (Network::Client::MQTTv5 object) to run the command from works great when running other methods (disconnect, publish, etc) from the separate thread. However, when I try to run eventLoop from the separathe thread, there seems to be an error.
When there are no messages/events to be handled, it looks like it gets stuck in the mbedtls_ssl_read function. I turned on MbedTLS debugging and it printed the following:

I (200617) mbedtls: ssl_tls.c:2813 in_left: 0, nb_want: 5

I (200617) mbedtls: ssl_tls.c:8389 => read

I (200627) mbedtls: ssl_tls.c:4419 => read record

I (200627) mbedtls: ssl_tls.c:2628 => fetch input

I (200637) mbedtls: ssl_tls.c:2789 in_left: 0, nb_want: 5

It prints this once every three seconds. If I change the socket connection to non-blocking, it prints this constantly.
Calling eventLoop() doesn't return in any errors.

When I call eventLoop and there are multiple publish messages/events to be handled, it handles a single event correctly, then disconnects from the (locally hosted) broker.

missing functions getEthernetRate and getWIFIRate

I am compiling on debian 9 and the linker does not find getEthernetRate() nor getWIFIRate(). I searched the includes and the libraries and the internet and found nothing.

Please where can I get those functions?

(I had to add link libraries pthread and dl to target MQTTc as well and lower the minimum cmake version to 3.8).

eMQTT5 does not install at all

Hi, it's me again ๐Ÿ˜†,

when including eMQTT5 into some project, which wants to install eMQTT5 along with some other product, at least the shared object should be installed, possibly even the test program. Using eMQTT5 without local modifications will lead to the effect, that the installed product will not have eMQTT5 installed. CMake before version 3.13 will not allow this to be added from the outside, so could you please add something like

install(TARGETS eMQTT5 LIBRARY DESTINATION lib)

and

install(TARGETS MQTTc RUNTIME DESTINATION bin)

to the respective project files?

read properties of received packet

I try to parse mqtt packets which I dumped with Wireshark using the MQTTParsePacket.cpp example. Works so far.
In my CONNECT packet, there is an UserProperty key value pair included. packet->dump() shows my value

I tried to read the Property but I was not sucesssful until now.

Using the example code for reading properties from [0], works not.

// packet is type of Protocol::MQTT::V5::ControlPacketSerializable*
Protocol::MQTT::V5::VisitorVariant visitor;
while(packet->props.getProperty(visitor)) {
     switch(visitor.propertyType()) {
     case Protocol::MQTT::V5::UserProperty:
     {
         // no code until now
         break;
     }
     }
 }

This does not compile because packet has no props member.

Next try:
explicit cast packet to Protocol::MQTT::V5::ConnectPacket*, result is:
no matching function for call to 'Protocol::MQTT::V5::Properties::getProperty(Protocol::MQTT::V5::VisitorVariant&)'

Next try:

Protocol::MQTT::V5::ConnectPacket* connect_packet = dynamic_cast<Protocol::MQTT::V5::ConnectPacket*>(packet);
auto myProp = connect_packet->props.getProperty(Protocol::MQTT::V5::UserProperty);
if(myProp->type == Protocol::MQTT::V5::UserProperty) {
   ????
}

Can I cast myProp to something like UserPropertyProp or do I need a completely different solution.

[0] https://github.com/X-Ryl669/eMQTT5/blob/master/doc/ClientAPI.md#receiving-packets-from-the-broker

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.