← Back to DevBytes
Developer Tutorial July 15, 2026 16 min read

MQTT Protocol: A Complete Reference Guide

What is MQTT?

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe network protocol that transports messages between devices. It was invented by Andy Stanford-Clark of IBM and Arlen Nipper of Cirrus Link in 1999 to monitor an oil pipeline over satellite connections. The protocol runs over TCP/IP and is designed for low-bandwidth, high-latency, or unreliable networks — exactly the conditions found in most IoT deployments.

At its core, MQTT follows a broker-based architecture. Clients do not communicate directly with each other. Instead, every message passes through a central server called a broker. A client that sends information is called a publisher. A client that receives information is called a subscriber. Publishers and subscribers never know about each other; they only know about topics — named logical channels that the broker uses to route messages.

Core Concepts

How the Protocol Works — The Wire Format

MQTT messages consist of a fixed header (mandatory, 2 bytes minimum), a variable header (optional), and a payload (optional). The fixed header encodes the message type, flags (including QoS level and retain flag), and remaining length. There are 14 message types in MQTT 3.1.1, including CONNECT, CONNACK, PUBLISH, PUBACK, SUBSCRIBE, SUBACK, UNSUBSCRIBE, UNSUBACK, PINGREQ, PINGRESP, and DISCONNECT.

Here is the structure of a PUBLISH packet's fixed header:

Byte 1:  | Message Type (4 bits) | DUP | QoS (2 bits) | RETAIN |
         |   0011 for PUBLISH    | flag|               |  flag  |

Byte 2+:  Remaining Length (variable-length encoding, 1-4 bytes)

The variable header for PUBLISH contains the topic name (UTF-8 encoded, preceded by a 2-byte length field) and, for QoS > 0, a Packet Identifier. The payload follows as raw bytes — MQTT is completely payload-agnostic. You can send JSON, binary protobuf, plain text, or even encrypted blobs.

Why MQTT Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

MQTT has become the de facto standard for IoT communication. Here's why it dominates this space:

Setting Up a Broker

The most popular open-source broker is Mosquitto, maintained by the Eclipse Foundation. You can install it on any Linux system or run it via Docker.

Installing Mosquitto on Ubuntu/Debian

sudo apt update
sudo apt install mosquitto mosquitto-clients -y
sudo systemctl enable mosquitto
sudo systemctl start mosquitto

To test locally, open two terminals. In the first, start a subscriber:

mosquitto_sub -t "test/topic" -v

In the second, publish a message:

mosquitto_pub -t "test/topic" -m "Hello, MQTT world!"

You should see test/topic Hello, MQTT world! appear in the subscriber terminal.

Running Mosquitto with Docker

docker run -d --name mosquitto \
  -p 1883:1883 \
  -p 9001:9001 \
  -v $(pwd)/mosquitto.conf:/mosquitto/config/mosquitto.conf \
  eclipse-mosquitto:latest

Port 1883 is the standard MQTT TCP port. Port 9001 is commonly used for MQTT over WebSockets (useful for browser-based clients).

Configuration File Example

A minimal mosquitto.conf with authentication:

# mosquitto.conf
listener 1883
allow_anonymous false

# Password file (create with mosquitto_passwd utility)
password_file /mosquitto/config/passwd

# Persistence for QoS 1/2 messages across restarts
persistence true
persistence_location /mosquitto/data/

# Logging
log_dest file /mosquitto/log/mosquitto.log
log_type error
log_type warning
log_type notice

Create a password file using the mosquitto_passwd utility:

mosquitto_passwd -b /path/to/passwd username password

MQTT Client Programming in Python

The paho-mqtt library is the standard Python client. Install it with:

pip install paho-mqtt

Basic Publisher

import paho.mqtt.client as mqtt
import time
import json

# Broker configuration
BROKER = "localhost"
PORT = 1883
TOPIC = "sensors/temperature"

# Create client instance
client = mqtt.Client(client_id="publisher_01")

# Set username/password if required
client.username_pw_set("user", "password")

# Connect to broker
client.connect(BROKER, PORT, keepalive=60)

# Publish a simple text message
client.publish(TOPIC, "23.5°C", qos=1)

# Publish JSON payload
payload = json.dumps({
    "sensor_id": "temp_001",
    "value": 23.5,
    "unit": "celsius",
    "timestamp": int(time.time())
})
client.publish(TOPIC, payload, qos=1, retain=False)

# Graceful disconnect
client.disconnect()

Basic Subscriber with Callbacks

import paho.mqtt.client as mqtt
import json

BROKER = "localhost"
PORT = 1883
TOPIC = "sensors/#"  # Wildcard: subscribe to all sensor subtopics

def on_connect(client, userdata, flags, rc):
    """Callback for broker connection acknowledgement."""
    rc_strings = {
        0: "Connection successful",
        1: "Connection refused - incorrect protocol version",
        2: "Connection refused - invalid client identifier",
        3: "Connection refused - server unavailable",
        4: "Connection refused - bad username or password",
        5: "Connection refused - not authorised"
    }
    print(f"Connection result: {rc_strings.get(rc, f'Unknown code {rc}')}")
    if rc == 0:
        # Subscribe to topic(s) after successful connection
        client.subscribe(TOPIC, qos=1)
        print(f"Subscribed to: {TOPIC}")

def on_message(client, userdata, msg):
    """Callback for incoming PUBLISH messages."""
    print(f"Topic: {msg.topic}")
    print(f"QoS: {msg.qos}")
    print(f"Retain flag: {msg.retain}")
    
    # Try to parse JSON, fall back to plain text
    try:
        data = json.loads(msg.payload.decode("utf-8"))
        print(f"Parsed JSON: {data}")
    except (json.JSONDecodeError, UnicodeDecodeError):
        print(f"Raw payload: {msg.payload}")

def on_disconnect(client, userdata, rc):
    """Callback for disconnection."""
    if rc != 0:
        print(f"Unexpected disconnection. RC: {rc}")

# Create client and bind callbacks
client = mqtt.Client(client_id="subscriber_01")
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect

# Optional: set credentials
client.username_pw_set("user", "password")

# Connect and start the network loop
client.connect(BROKER, PORT, keepalive=60)

# Blocking loop - processes network traffic and dispatches callbacks
# Use loop_forever() for a blocking main loop, or loop_start() for background threading
client.loop_forever()

Using loop_start() for Non-Blocking Operation

import paho.mqtt.client as mqtt
import time
import signal
import sys

client = mqtt.Client(client_id="nonblocking_client")
client.connect("localhost", 1883)

# Start network loop in background thread
client.loop_start()

# Subscribe and publish while your main thread does other work
client.subscribe("commands/#", qos=2)

for i in range(10):
    client.publish("status/loop", f"Iteration {i}", qos=1)
    time.sleep(1)

# Stop the background loop cleanly
client.loop_stop()
client.disconnect()

MQTT QoS Deep Dive

Understanding QoS is critical for building reliable MQTT systems. Here's what each level means in practice:

QoS 0 — At Most Once (Fire and Forget)

The message is sent once and not acknowledged. If the subscriber is offline or the network drops the packet, the message is lost. This is the fastest option and works well for high-frequency telemetry where occasional data loss is acceptable (e.g., temperature readings every second).

# QoS 0 publish — no ack expected, no retry
client.publish("sensors/noise", "42dB", qos=0)

QoS 1 — At Least Once (Acknowledged Delivery)

The sender stores the message until it receives a PUBACK from the broker. If no PUBACK arrives within a timeout, the message is retransmitted with the DUP flag set. Subscribers may receive duplicates — your application must be idempotent.

# QoS 1 publish — expects PUBACK from broker
msg_info = client.publish("alerts/warning", "High temperature!", qos=1)
msg_info.wait_for_publish()  # Block until broker acknowledges
print(f"Published with mid={msg_info.mid}")

QoS 2 — Exactly Once (Four-Way Handshake)

This is the highest level and involves a four-step handshake: PUBLISH → PUBREC → PUBREL → PUBCOMP. Both sender and receiver persist state until the exchange completes. No duplicates, no loss — but higher latency and overhead. Use this for mission-critical commands like opening a valve or dispensing medication.

# QoS 2 publish — full four-way handshake
client.publish("critical/shutdown", "EMERGENCY_STOP", qos=2)

Important nuance: QoS in MQTT is per message and per hop. The QoS between publisher and broker can differ from the QoS between broker and subscriber. The broker downgrades QoS if a subscriber requested a lower QoS on its subscription. For example, if a publisher sends with QoS 2 but a subscriber subscribed with QoS 0, the broker delivers to that subscriber with QoS 0.

MQTT Topics — Design Patterns and Wildcards

Topic design is one of the most consequential decisions in an MQTT system. A well-designed topic hierarchy makes filtering, access control, and debugging straightforward.

Topic Structure Best Practices

# Good topic naming: hierarchical, specific-to-general
factory/plant_a/line_3/machine_7/temperature
factory/plant_a/line_3/machine_7/pressure
factory/plant_a/line_3/machine_7/status

# Use singular nouns, lowercase, no spaces
home/bedroom/lamp/set
home/bedroom/lamp/state

# Version your topics for backward compatibility
device/firmware/v2/update
device/firmware/v2/response

Wildcard Subscriptions

# Subscribe to temperature from all machines on all lines in all plants
client.subscribe("factory/+/+/+/temperature", qos=1)

# Subscribe to everything under plant_a (including subtopics)
client.subscribe("factory/plant_a/#", qos=1)

# Note: # cannot be used mid-topic — this is INVALID:
# client.subscribe("factory/#/temperature")  # WILL NOT WORK

Avoid Leading Slashes

MQTT topics do not need a leading forward slash. Using one creates an empty first level that complicates wildcard matching and is discouraged by the specification.

# Good
sensors/outdoor/humidity

# Avoid
/sensors/outdoor/humidity

Retained Messages and LWT

Retained Messages

When a publisher sets the retain flag, the broker stores the message alongside the topic. Every new subscriber to that topic immediately receives the last retained message — before any new live messages arrive. This is perfect for publishing static or slowly-changing state like device configuration, sensor metadata, or last-known values.

# Publish a retained message — new subscribers get this immediately
client.publish("device/thermostat/target_temp", "22", qos=1, retain=True)

# Clear a retained message by publishing a zero-byte payload with retain=True
client.publish("device/thermostat/target_temp", "", qos=1, retain=True)

Last Will and Testament (LWT)

LWT is a message that the broker publishes automatically if the client disconnects ungracefully (network failure, power loss, or keep-alive timeout). It's set during the CONNECT phase and cannot be changed during the session.

# Set LWT during client initialization
client = mqtt.Client(client_id="sensor_01")

# When this client disconnects unexpectedly, the broker publishes:
#   "sensor_01/status" with payload "OFFLINE" at QoS 1, retained
client.will_set(
    topic="sensor_01/status",
    payload="OFFLINE",
    qos=1,
    retain=True
)

# On graceful connection, publish ONLINE so subscribers know you're up
client.connect("localhost", 1883)
client.publish("sensor_01/status", "ONLINE", qos=1, retain=True)

Persistent Sessions and Clean Start

In MQTT 5, the cleanStart flag replaces MQTT 3.1.1's cleanSession. When cleanStart=False (or clean_session=False in older clients), the broker remembers:

When the client reconnects, the broker replays queued messages and reinstates subscriptions automatically. This is essential for unreliable mobile or satellite links.

# Python paho-mqtt persistent session example
client = mqtt.Client(
    client_id="persistent_sensor_01",
    clean_session=False  # Broker remembers subscriptions and queues
)

# Connect and subscribe
client.connect("localhost", 1883)
client.subscribe("commands/#", qos=2)

# Even if the client disconnects and reconnects later,
# QoS 2 messages sent to commands/# while offline will be delivered
client.loop_forever()

MQTT 5 — What's New

MQTT 5, released in 2019, adds significant improvements while maintaining backward compatibility with MQTT 3.1.1. Key additions include:

# MQTT 5 example with paho-mqtt v2
import paho.mqtt.client as mqtt
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes

client = mqtt.Client(
    client_id="mqtt5_client",
    protocol=mqtt.MQTTv5
)

# Set session expiry to 3600 seconds (1 hour)
properties = Properties(PacketTypes.CONNECT)
properties.SessionExpiryInterval = 3600

client.connect("localhost", 1883, clean_start=mqtt.MQTT_CLEAN_START_FIRST)

# Publish with message expiry and user properties
pub_props = Properties(PacketTypes.PUBLISH)
pub_props.MessageExpiryInterval = 300  # Expire after 5 minutes if not delivered
pub_props.UserProperty = [("source", "backend"), ("priority", "high")]

client.publish(
    "events/alerts",
    "Server restart required",
    qos=2,
    properties=pub_props
)
client.loop_forever()

Security Considerations

A production MQTT deployment requires multiple layers of security:

TLS/SSL Encryption

Always use TLS in production. Plain-text MQTT over port 1883 exposes credentials and payloads to anyone on the network path.

# TLS-secured MQTT client
import ssl

client = mqtt.Client(client_id="secure_client")
client.tls_set(
    ca_certs="/etc/mosquitto/certs/ca.crt",      # CA certificate
    certfile="/etc/mosquitto/certs/client.crt",   # Client certificate
    keyfile="/etc/mosquitto/certs/client.key",    # Client private key
    cert_reqs=ssl.CERT_REQUIRED,                  # Require broker certificate verification
    tls_version=ssl.PROTOCOL_TLSv1_2
)

client.username_pw_set("user", "password")
client.connect("mqtt.example.com", 8883)  # Port 8883 is standard MQTT+TLS
client.loop_forever()

Broker-Side ACLs

Mosquitto supports access control lists to restrict which clients can publish or subscribe to which topics. Create an ACL file:

# /etc/mosquitto/acl.conf
# User 'sensor_temp' can only publish to its own sensor topic
user sensor_temp
topic write sensors/temperature/#
topic read sensors/temperature/#

# User 'dashboard' can only read all sensor data
user dashboard
topic read sensors/#
topic deny write #

# Admin has full access
user admin
topic readwrite #

Reference this ACL file in mosquitto.conf:

acl_file /etc/mosquitto/acl.conf

Additional Security Practices

Testing and Debugging MQTT

mosquitto_sub with Verbose Output

# Monitor all traffic on the broker (requires wildcard subscription access)
mosquitto_sub -t "#" -v -d

# -v shows topic alongside payload
# -d enables debug output showing message IDs, QoS, and retain flags

MQTT Explorer (GUI Tool)

MQTT Explorer is an excellent graphical client for visualizing topic trees, retained messages, and real-time traffic. It runs on Windows, macOS, and Linux. Connect it to your broker and browse the entire topic namespace interactively.

Python Debug Client

import paho.mqtt.client as mqtt

def on_message(client, userdata, msg):
    print(f"\n--- Message Received ---")
    print(f"Topic:      {msg.topic}")
    print(f"Payload:    {msg.payload}")
    print(f"QoS:        {msg.qos}")
    print(f"Retained:   {msg.retain}")
    print(f"Mid:        {msg.mid}")
    print(f"Timestamp:  {msg.timestamp}")

client = mqtt.Client(client_id="debug_listener")
client.on_message = on_message

# Enable full debug logging
import logging
logging.basicConfig(level=logging.DEBUG)

client.connect("localhost", 1883)
client.subscribe("#", qos=2)
client.loop_forever()

Common Pitfalls and How to Avoid Them

# Example: Offloading message processing to avoid blocking
import threading
import queue

work_queue = queue.Queue()

def on_message(client, userdata, msg):
    # Immediately enqueue; callback returns quickly
    work_queue.put((msg.topic, msg.payload))

def worker():
    while True:
        topic, payload = work_queue.get()
        # Perform time-consuming processing here
        process_complex_data(topic, payload)
        work_queue.task_done()

# Start worker threads
for _ in range(4):
    threading.Thread(target=worker, daemon=True).start()

# Now connect and subscribe — on_message stays fast
client = mqtt.Client()
client.on_message = on_message
client.connect("localhost", 1883)
client.subscribe("#", qos=1)
client.loop_forever()

Scalability Patterns

Bridging Brokers

Mosquitto supports broker bridging — connecting two MQTT brokers so that messages published on one are forwarded to the other. This enables geographic distribution and high availability.

# Bridge configuration in mosquitto.conf
connection cloud-bridge
address mqtt.cloud-provider.com:8883
remote_username bridge_user
remote_password bridge_password
topic sensors/# both 1
bridge_cafile /etc/mosquitto/certs/ca.crt
bridge_insecure false
try_private true
start_type automatic
cleansession false

Horizontal Scaling with EMQX or HiveMQ

For truly massive deployments (millions of concurrent connections), consider clustered brokers like EMQX or HiveMQ. These use distributed erlang-based or Kubernetes-native architectures to share topic trees and session state across nodes while presenting a single logical broker to clients.

Load Balancing

Use TCP load balancers (HAProxy, NGINX Stream module) in front of multiple broker instances. Configure sticky sessions based on client ID to ensure persistent session state remains on the same broker node.

# Example NGINX stream configuration for MQTT load balancing
stream {
    upstream mqtt_backend {
        hash $remote_addr consistent;
        server broker1:1883;
        server broker2:1883;
        server broker3:1883;
    }
    server {
        listen 1883;
        proxy_pass mqtt_backend;
        proxy_timeout 300s;
    }
}

MQTT in Web Browsers

Browsers cannot open raw TCP sockets, but MQTT over WebSocket (typically on port 9001) bridges this gap. The JavaScript library mqtt.js is the standard choice.

<!-- Browser-based MQTT client using mqtt.js -->
<script src="https://unpkg.com/mqtt/dist/mqtt.min.js"></script>
<script>
  // Connect via WebSocket
  const client = mqtt.connect('ws://localhost:9001', {
    clientId: 'web_client_' + Math.random().toString(16).substr(2, 8),
    username: 'user',
    password: 'password'
  });

  client.on('connect', () => {
    console.log('Connected to broker');
    client.subscribe('sensors/#', { qos: 1 });
  });

  client.on('message', (topic, message) => {
    console.log(`Topic: ${topic}`);
    console.log(`Payload: ${message.toString()}`);
  });

  // Publish from browser
  document.getElementById('sendBtn').addEventListener('click', () => {
    client.publish('commands/led', 'TOGGLE', { qos: 1 });
  });
</script>

Best Practices Summary

Conclusion

MQTT has earned its place as the backbone of IoT communication through a combination of simplicity, efficiency, and reliability. Its publish-subscribe model decouples producers and consumers, its minimal wire overhead conserves bandwidth and battery life, and its QoS system provides fine-grained control over message delivery guarantees. From a single Raspberry Pi monitoring a greenhouse to fleets of millions of connected vehicles, MQTT scales across the entire spectrum of connected systems.

The protocol's evolution from MQTT 3.1.1 to MQTT 5 brings modern features like session expiry, flow control, and user properties while preserving the lightweight spirit that made it successful. By following the topic design principles, security practices, and code patterns outlined in this guide, you can build MQTT systems that are robust, maintainable, and ready for production. Whether you're publishing sensor data from a microcontroller, building a real-time dashboard in a browser, or orchestrating commands across a distributed industrial system, MQTT provides the reliable messaging foundation you need.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles