Bridging Raw TCP Protocols to MQTT Topics

Overview Link to heading

The rawtcp_to_mqtt_bridge project provides a robust and flexible solution for connecting devices or systems that communicate via a generic binary TCP protocol to modern IoT infrastructures based on MQTT. This bridge enables seamless integration between legacy or custom devices and MQTT brokers, facilitating data exchange, monitoring, and control in distributed environments.

Purpose Link to heading

Many industrial, embedded, or custom devices use simple TCP protocols to send and receive data. However, the rapid adoption of MQTT as a standard in modern IoT ecosystems creates the need to interconnect these two worlds. This project solves that problem by converting TCP data streams into MQTT messages and vice versa, allowing legacy devices to participate in MQTT-based systems without firmware or hardware modifications.

The Need for Lightweight Protocol Bridges in IoT Link to heading

As IoT expands, many devices—especially those deployed in the field or industrial settings—must operate with limited resources and restricted networks. Cellular networks, for example, are often the only option in remote sites and require using extremely simple binary protocols to save bandwidth, energy, and complexity.

Why Use Simple Protocols? Link to heading

• Bandwidth Efficiency: Complex protocols increase traffic and cost on metered networks.
• Hardware Limitations: Many embedded devices have little memory and CPU.
• Energy Savings: Short, simple messages allow radios to be active less time, extending device life.
• Simplicity: Raw TCP with basic framing (like SLIP) is easy to implement even on modest microcontrollers.

The Integration Challenge Link to heading

These binary protocols, while efficient, are incompatible with modern cloud platforms that expect standardized, rich interactions like MQTT. MQTT enables publish/subscribe semantics, QoS, retention, and easy integration with dashboards and automation.

Bridging TCP and MQTT is crucial for:

• Modernizing legacy systems without costly firmware changes.
• Scaling cloud-connected solutions on top of existing devices.
• Enabling remote monitoring, control, and analytics using mature MQTT tools.

Architecture Diagram Link to heading

Below is a diagram illustrating the architecture and data flow:

Legend:

• Devices use lightweight binary TCP protocols, with SLIP framing for robustness.
• Data crosses potentially unreliable or costly networks (cellular, satellite).
• The bridge receives SLIP-framed data, decodes it, and publishes MQTT messages.
• The MQTT broker distributes messages to dashboards, applications, or cloud services.

SLIP: Framing and Robustness in Heterogeneous Networks Link to heading

SLIP (Serial Line Internet Protocol) is used to delimit packets on the TCP stream. It is extremely simple, efficient, and enables automatic resynchronization if bytes are lost—ideal for unreliable links.

SLIP Framing Example:

+-----+-----------+-----+-----------+-----+-----------+
| END | PAYLOAD 1 | END | PAYLOAD 2 | END | PAYLOAD 3 |
+-----+-----------+-----+-----------+-----+-----------+

• Each payload ends with a special END byte (0xC0).
• If the END is lost, the receiver waits for the next one to resynchronize.

SLIP Resynchronization Visualization:

Features and Capabilities Link to heading

• TCP to MQTT Bridging: Listens for raw TCP connections and forwards the data as MQTT messages to configurable topics.
• Protocol Agnostic: Can work with any device or software that uses a basic TCP stream.
• SLIP Framing: Uses SLIP for packet delimitation, enabling automatic resynchronization and robustness.
• Configurability: Customize TCP ports, MQTT broker, topic mapping, and more using YAML files.
• Multi-Client Support: Handles multiple simultaneous TCP connections.
• Cross-Platform: Developed mainly in C++, with auxiliary scripts in Python and CMake.

Project Structure Link to heading

rawtcp_to_mqtt_bridge/
├── src/        # Core C++ code (TCP, SLIP, MQTT)
├── include/    # Headers for modularity
├── python/     # Auxiliary scripts for testing/configuration
├── CMakeLists.txt  # Build configuration
└── README.md   # Documentation

Specifying the Binary TCP Protocol and Mapping to MQTT Templates Link to heading

A core feature of rawtcp_to_mqtt_bridge is the ability to flexibly define binary TCP packet structures and transform them into MQTT messages. This is achieved by describing packet formats in YAML files, allowing the bridge to parse binary streams and publish them into MQTT topics according to user-defined templates.

How to Specify Binary TCP Protocols Link to heading

Packet structures are specified in YAML files within the packets/ directory (see the packet_defs section in your config.yaml). Each YAML file defines one or more packet types, including binary layout, field types, offsets, and the corresponding MQTT topic and payload format.

Example: Defining a Sensor Data Packet Link to heading

sensor_data:
  mqtt:
    topic: "sensors/data_sensor_{{sensor_id}}"
    payload: |
      {
        "temperature": {{temperature}},
        "humidity": {{humidity}},
        "pressure": {{pressure}}
      }      
  fields:
    - name: packet_type
      type: uint8
      offset: 0
      value: 0x10  # Identifier

    - name: sensor_id
      type: uint16
      offset: 1

    - name: temperature
      type: float32
      offset: 3

    - name: humidity
      type: float32
      offset: 7

    - name: pressure
      type: float32
      offset: 11

Explanation:

• Each field has a name, type, and offset in the binary packet.
• The packet_type field acts as an identifier (here, 0x10).
• The mqtt section defines:
  • The topic template (topics can use field values, e.g., sensor_id)
  • The payload template (typically JSON, rendered with field values)

Example: Binary Packet and MQTT Mapping Link to heading

If the device sends these bytes (big-endian):

The bridge:

• Recognizes packet_type as 0x10 (sensor_data).
• Parses all fields by type and offset.
• Publishes:

Topic: sensors/data_sensor_1

Payload:

{
  "temperature": 10.0,
  "humidity": 50.0,
  "pressure": 60.0
}

Example: Bitfields and Arrays Link to heading

You can define advanced structures like bitfields (flags) or arrays for more complex layouts:

device_status:
  mqtt:
    topic: "devices/status/{{device_id}}"
    payload: |
      {
        "flags": {{flags}},
        "voltage": {{voltage}}
      }      
  fields:
    - name: packet_type
      type: uint8
      offset: 0
      value: 0x20

    - name: device_id
      type: uint8
      offset: 1

    - name: flags
      type: uint8
      offset: 2
      bitfield:
        bit_offset: 0
        bit_count: 8

    - name: voltage
      type: float32
      offset: 3

How the Bridge Uses the Template Link to heading

• The bridge identifies packets by their unique identifier field (value in fields, e.g., packet_type: 0x10).
• It extracts each field based on type and offset, supporting types like uint8, uint16, float32, and even arrays.
• The extracted values are injected into the MQTT topic and payload templates using the inja template engine.
• This enables dynamic MQTT topics and rich JSON payloads, making it easy to integrate with any MQTT-based backend.

More on Packet Definitions Link to heading

• All packet definition YAMLs are loaded from the paths and patterns configured in config.yaml under packet_defs.
• You can group definitions by type (sensors, commands, status) to scale and organize.

For more examples, see the README.md on GitHub.

Problem Solved Link to heading

Connecting legacy or proprietary devices to modern IoT platforms is a frequent challenge in industrial automation, smart buildings, and research. Many lack native MQTT or publish/subscribe support. This bridge eliminates the need for custom firmware/middleware, speeding up integration and reducing complexity. It enables modernization and unlocks the power of cloud analytics and management.

In scenarios where data usage matters (cellular, satellite), the use of simple binary protocols and SLIP framing ensures efficiency and robustness. The bridge not only translates between worlds but also adds resilience, ideal for variable environments.

Use Cases Link to heading

• Industrial equipment: Sensors send data via TCP and reach the cloud via MQTT.
• Home automation: TCP devices can interact with MQTT controllers.
• Prototyping: Developers can use simple TCP and later integrate into MQTT infrastructure.
• Field IoT: Benefit from SLIP resynchronization for intermittent links.
• Modernization: Integrate legacy devices into dashboards and cloud services without firmware changes.

Data Flow Example Link to heading

Possible Improvements Link to heading

Although the bridge already provides essential capabilities, there are many ways to enhance it:

• Encryption:: TLS/SSL for TCP and MQTT, securing communication over public networks.
• Authentication:: Mechanisms to authenticate TCP clients and limit unauthorized access.
• Compression:: Configurable compression (e.g., zlib) to further reduce data usage.
• Bidirectional MQTT→TCP Communication:: Improved support for sending commands/config from MQTT to TCP devices.
• Protocol Translation and Validation:: Advanced parsing/validation for intelligent bridging.
• Advanced Monitoring and Logging:: Integration with monitoring systems and detailed logging for troubleshooting.
• High Availability and Scalability:: Support for redundant deployment, clustering, or load balancing.

Conclusion Link to heading

rawtcp_to_mqtt_bridge is a practical, open-source tool that connects the legacy world of binary TCP with the modern, scalable world of MQTT. Its focus on efficiency, robustness, and flexibility makes it ideal for modernizing installations and enabling data-driven applications even in the most demanding environments.

As the IoT ecosystem evolves, bridges like this will be critical to integrate the old with the new, extending the life and value of existing assets, and enabling truly connected, intelligent solutions.