← Back to DevBytes

XMPP Protocol: A Complete Reference Guide

Introduction to XMPP

XMPP, which stands for Extensible Messaging and Presence Protocol, is an open XML-based communication protocol designed for real-time messaging, presence information, and contact list maintenance. Originally developed in 1999 by Jeremie Miller as the Jabber protocol, XMPP has evolved into a mature, decentralized, and highly extensible standard maintained by the XMPP Standards Foundation (XSF).

What is XMPP?

At its core, XMPP is a protocol for streaming XML elements between network endpoints. Unlike many modern messaging systems that rely on centralized servers controlled by a single company, XMPP uses a federated architecture similar to email. Anyone can run an XMPP server, and servers can communicate with each other, allowing users on different servers to exchange messages seamlessly.

The protocol operates over TCP connections and uses XML streams to exchange structured data. Each connection establishes a long-lived XML stream from client to server, and another stream from server to client. Within these streams, individual XML stanzas are sent back and forth to carry out operations like authentication, messaging, and presence updates.

Why XMPP Matters

XMPP remains relevant in modern software development for several compelling reasons:

XMPP Architecture

Understanding the XMPP architecture is essential before building any application on top of it. The protocol defines a client-server model with federation between servers.

Core Components

The XMPP ecosystem consists of several key components that work together:

Addressing Scheme

Every entity in XMPP is identified by a Jabber ID (JID), which follows a format similar to an email address. Understanding JID structure is fundamental:

user@domain.tld/resource

The JID has three parts:

A JID without a resource part is called a "bare JID" and identifies a user account. A JID with a resource part is a "full JID" and identifies a specific connected session. This distinction matters when routing messages, as sending to a bare JID lets the server decide which session receives the message, while sending to a full JID targets a specific session.

Getting Started with XMPP

To begin developing with XMPP, you need access to an XMPP server and a client library for your programming language of choice.

Setting up a Server

For development purposes, Prosody is an excellent choice due to its simplicity and Lua-based configuration. Here is a basic Prosody configuration file:

-- /etc/prosody/prosody.cfg.lua

-- Server-wide settings
admins = { "admin@example.com" }

-- Enable modules
modules_enabled = {
    "roster";        -- Contact list management
    "saslauth";      -- Authentication
    "tls";           -- Encryption
    "dialback";      -- Server-to-server authentication
    "disco";         -- Service discovery
    "ping";          -- Keep-alive
    "pep";           -- Personal eventing
    "carbons";       -- Message carbons
    "mam";           -- Message archive management
    "muc";           -- Multi-user chat
    "pubsub";        -- Publish-subscribe
}

-- Virtual hosts
VirtualHost "example.com"
    enabled = true
    ssl = {
        key = "/etc/prosody/certs/example.com.key";
        certificate = "/etc/prosody/certs/example.com.crt";
    }

-- Component for group chat
Component "conference.example.com" "muc"

-- Component for PubSub
Component "pubsub.example.com" "pubsub"

After saving the configuration, restart Prosody and create a test user:

prosodyctl adduser alice@example.com
prosodyctl adduser bob@example.com

Client Libraries

XMPP client libraries exist for virtually every programming language. Some popular options include:

Connecting and Authenticating

The first step in any XMPP application is establishing a connection and authenticating. The connection process involves opening an XML stream, negotiating TLS, and authenticating via SASL. Here is a Python example using Slixmpp:

import asyncio
import slixmpp
from slixmpp.xmlstream import ET

class XMPPClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.add_event_handler("session_start", self.session_start)
        self.add_event_handler("message", self.message_handler)
        self.add_event_handler("changed_status", self.presence_handler)

    async def session_start(self, event):
        # Send initial presence
        self.send_presence()
        # Request the roster (contact list)
        await self.get_roster()
        print("Session started and presence sent!")

    def message_handler(self, msg):
        if msg['type'] in ('chat', 'normal'):
            print(f"Message from {msg['from']}: {msg['body']}")

    def presence_handler(self, presence):
        print(f"Presence update: {presence['from']} is {presence['type'] or 'available'}")

async def main():
    client = XMPPClient("alice@example.com", "password")
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

For JavaScript developers, the @xmpp/client package provides a modern, promise-based API:

const { client, xml } = require("@xmpp/client");
const debug = require("@xmpp/debug");

const xmpp = client({
  service: "xmpp://example.com:5222",
  domain: "example.com",
  username: "alice",
  password: "password",
});

debug(xmpp, true);

xmpp.on("error", (err) => {
  console.error("XMPP error:", err);
});

xmpp.on("offline", () => {
  console.log("Client is offline");
});

xmpp.on("stanza", async (stanza) => {
  console.log("Received stanza:", stanza.toString());
});

xmpp.on("online", async (address) => {
  console.log("Connected as", address.toString());

  // Send initial presence
  await xmpp.send(xml("presence"));

  // Send a message
  const message = xml(
    "message",
    { type: "chat", to: "bob@example.com" },
    xml("body", {}, "Hello Bob from Node.js!")
  );
  await xmpp.send(message);
});

xmpp.start().catch(console.error);

Under the hood, the connection process follows this XML exchange. The client opens a stream:

<?xml version='1.0'?>
<stream:stream
    xmlns='jabber:client'
    xmlns:stream='http://etherx.jabber.org/streams'
    to='example.com'
    version='1.0'>

The server responds with its stream and available authentication mechanisms:

<stream:stream
    xmlns='jabber:client'
    xmlns:stream='http://etherx.jabber.org/streams'
    from='example.com'
    id='abc123'
    version='1.0'>
<stream:features>
  <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'>
    <required/>
  </starttls>
  <mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>
    <mechanism>SCRAM-SHA-256</mechanism>
    <mechanism>PLAIN</mechanism>
  </mechanisms>
</stream:features>

The Three Core Stanza Types

XMPP defines three fundamental stanza types that form the basis of all communication. Understanding these stanzas is critical for any XMPP developer.

Message Stanzas

Message stanzas are used for one-to-one communication, group chat, and notifications. They are "fire and forget" by default, meaning the server does not confirm delivery unless explicitly requested:

<message
    from='alice@example.com/desktop'
    to='bob@example.com'
    type='chat'
    id='msg001'>
  <body>Hello Bob, how are you?</body>
  <active xmlns='http://jabber.org/protocol/chatstates'/>
</message>

The type attribute on a message stanza determines its context:

Presence Stanzas

Presence stanzas communicate availability status. They are the mechanism that makes XMPP a "presence protocol" rather than just a messaging protocol:

<presence from='alice@example.com/desktop'>
  <show>away</show>
  <status>In a meeting</status>
  <priority>10</priority>
</presence>

The show element can contain one of four values: away, chat (available for chat), dnd (do not disturb), or xa (extended away). The status element holds a human-readable status message, and priority determines which resource receives messages sent to the bare JID.

IQ Stanzas

IQ (Info/Query) stanzas implement a request-response pattern, similar to HTTP. They always require an id attribute for matching responses and use a type attribute of get, set, result, or error:

<!-- Request: Get Bob's version info -->
<iq from='alice@example.com/desktop'
    to='bob@example.com/laptop'
    type='get'
    id='version1'>
  <query xmlns='jabber:iq:version'/>
</iq>

<!-- Response -->
<iq from='bob@example.com/laptop'
    to='alice@example.com/desktop'
    type='result'
    id='version1'>
  <query xmlns='jabber:iq:version'>
    <name>MyCustomClient</name>
    <version>1.0.0</version>
    <os>Linux 5.15.0</os>
  </query>
</iq>

Working with the Roster

The roster is XMPP's contact list system. Managing the roster involves IQ stanzas for retrieving, adding, and removing contacts, combined with presence subscriptions for seeing contacts' availability.

import slixmpp
from slixmpp.exceptions import IqError, IqTimeout

class RosterManager(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.add_event_handler("session_start", self.start)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Add a new contact
        await self.add_contact("charlie@example.com", "Charlie from work")

        # Retrieve and display the roster
        roster = self.client_roster
        for jid in roster:
            print(f"Contact: {jid}")
            print(f"  Name: {roster[jid]['name']}")
            print(f"  Subscription: {roster[jid]['subscription']}")
            print(f"  Groups: {roster[jid]['groups']}")

    async def add_contact(self, jid, name=None):
        """Add a contact to the roster and send a subscription request."""
        self.send_presence(pto=jid, ptype="subscribe")
        self.update_roster(jid, name=name)
        print(f"Subscription request sent to {jid}")

    async def remove_contact(self, jid):
        """Remove a contact from the roster."""
        self.send_presence(pto=jid, ptype="unsubscribe")
        self.send_presence(pto=jid, ptype="unsubscribed")
        await self.del_roster_item(jid)
        print(f"Removed {jid} from roster")

The subscription model in XMPP is bidirectional. When Alice wants to see Bob's presence, she sends a subscribe request. Bob must approve it with a subscribed response. If Bob also wants to see Alice's presence, he sends his own subscribe request. The subscription states are:

Messaging

One-to-One Chat

Sending a direct message is straightforward. Here is a complete example that sends a message and handles replies:

import asyncio
import slixmpp
from slixmpp.xmlstream import ET

class ChatClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password, recipient, message):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.recipient = recipient
        self.message = message
        self.add_event_handler("session_start", self.start)
        self.add_event_handler("message", self.on_message)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Send the message
        self.send_message(
            mto=self.recipient,
            mbody=self.message,
            mtype='chat'
        )
        print(f"Message sent to {self.recipient}")

    def on_message(self, msg):
        if msg['type'] in ('chat', 'normal'):
            print(f"\nReply from {msg['from'].bare}:")
            print(f"  {msg['body']}")
            
            # Handle chat state notifications
            if msg['chat_state']:
                print(f"  [Chat state: {msg['chat_state']}]")

async def main():
    client = ChatClient(
        "alice@example.com",
        "password",
        "bob@example.com",
        "Hey Bob! This is a test message."
    )
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

Group Chat (MUC)

Multi-User Chat (MUC), defined in XEP-0045, enables group chat rooms. Joining and participating in a MUC room requires specific stanza handling:

import asyncio
import slixmpp
from slixmpp.exceptions import MucError

class MUCClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password, room, nick):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.room = room
        self.nick = nick
        self.add_event_handler("session_start", self.start)
        self.add_event_handler("groupchat_message", self.on_muc_message)
        self.add_event_handler("muc::%s::got_online" % room, self.on_user_join)
        self.add_event_handler("muc::%s::got_offline" % room, self.on_user_leave)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Join the MUC room
        await self.plugin['xep_0045'].join_muc(
            self.room,
            self.nick,
            maxhistory="20"
        )
        print(f"Joined room: {self.room} as {self.nick}")

    def on_muc_message(self, msg):
        # Ignore messages from ourselves
        if msg['mucnick'] == self.nick:
            return
        print(f"[{msg['from'].resource}] {msg['body']}")

    def on_user_join(self, presence):
        print(f"*** {presence['muc']['nick']} joined the room")

    def on_user_leave(self, presence):
        print(f"*** {presence['muc']['nick']} left the room")

    async def send_to_room(self, message):
        self.send_message(
            mto=self.room,
            mbody=message,
            mtype='groupchat'
        )

async def main():
    client = MUCClient(
        "alice@example.com",
        "password",
        "devteam@conference.example.com",
        "AliceDev"
    )
    await client.connect()
    
    # Send a message after joining
    await asyncio.sleep(2)
    await client.send_to_room("Hello everyone in the dev team!")
    
    await client.process(forever=False)

asyncio.run(main())

The raw XML for joining a MUC room looks like this:

<presence
    from='alice@example.com/desktop'
    to='devteam@conference.example.com/AliceDev'>
  <x xmlns='http://jabber.org/protocol/muc'>
    <history maxchars='0'/>
  </x>
</presence>

Service Discovery

Service Discovery (XEP-0030) allows clients and components to discover what features and services an entity supports. This is how XMPP achieves its extensibility — clients can dynamically detect server capabilities:

import asyncio
import slixmpp

class DiscoveryClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.add_event_handler("session_start", self.start)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Discover server features
        print("=== Server Features ===")
        try:
            info = await self['xep_0030'].get_info(jid="example.com")
            for feature in info['disco_info']['features']:
                print(f"  Feature: {feature}")
            for identity in info['disco_info']['identities']:
                print(f"  Identity: {identity}")
        except Exception as e:
            print(f"Discovery error: {e}")

        # Discover available services (MUC, PubSub, etc.)
        print("\n=== Server Items ===")
        try:
            items = await self['xep_0030'].get_items(jid="example.com")
            for item in items['disco_items']['items']:
                print(f"  Service: {item[0]}")
                
                # Discover features of each service
                sub_info = await self['xep_0030'].get_info(jid=item[0])
                for identity in sub_info['disco_info']['identities']:
                    print(f"    Type: {identity['type']}, Name: {identity['name']}")
        except Exception as e:
            print(f"Items error: {e}")

async def main():
    client = DiscoveryClient("alice@example.com", "password")
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

The underlying IQ stanza for service discovery is:

<iq from='alice@example.com/desktop'
    to='example.com'
    type='get'
    id='disco1'>
  <query xmlns='http://jabber.org/protocol/disco#info'/>
</iq>

Publish-Subscribe (PubSub)

XMPP PubSub (XEP-0060) provides a publish-subscribe pattern that enables many advanced features like news feeds, avatar distribution, and event notifications. A node is created on the PubSub service, publishers push items to it, and subscribers receive notifications:

import asyncio
import slixmpp
from slixmpp.xmlstream import ET

class PubSubClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.pubsub_server = "pubsub.example.com"
        self.add_event_handler("session_start", self.start)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        node_name = "urn:example:notifications"

        # Create a node
        try:
            await self['xep_0060'].create_node(
                self.pubsub_server,
                node_name
            )
            print(f"Created node: {node_name}")
        except Exception as e:
            print(f"Node may already exist: {e}")

        # Subscribe to the node
        try:
            await self['xep_0060'].subscribe(
                self.pubsub_server,
                node_name
            )
            print(f"Subscribed to node: {node_name}")
        except Exception as e:
            print(f"Subscription error: {e}")

        # Publish an item
        payload = ET.Element("{urn:example:notifications}event")
        title = ET.SubElement(payload, "title")
        title.text = "System Update"
        desc = ET.SubElement(payload, "description")
        desc.text = "Maintenance scheduled for tonight at 2 AM UTC."

        try:
            result = await self['xep_0060'].publish(
                self.pubsub_server,
                node_name,
                payload=payload,
                id="event-001"
            )
            print(f"Published item: {result}")
        except Exception as e:
            print(f"Publish error: {e}")

        # Retrieve published items
        try:
            items = await self['xep_0060'].get_items(
                self.pubsub_server,
                node_name,
                max_items=10
            )
            print(f"\nRetrieved items:")
            for item in items['pubsub']['items']:
                print(f"  Item ID: {item['id']}")
        except Exception as e:
            print(f"Retrieve error: {e}")

async def main():
    client = PubSubClient("alice@example.com", "password")
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

File Transfer

File transfer in XMPP is handled through XEP-0096 (SI File Transfer) combined with XEP-0065 (SOCKS5 Bytestreams) or XEP-0047 (In-Band Bytestreams). Here is an example using In-Band Bytestreams, which works through the existing XMPP stream:

import asyncio
import slixmpp

class FileTransferClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password, recipient, filepath):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.recipient = recipient
        self.filepath = filepath
        self.add_event_handler("session_start", self.start)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Send a file using In-Band Bytestreams
        try:
            await self['xep_0096'].send_file(
                self.recipient,
                self.filepath,
                description="Project documentation"
            )
            print(f"File sent: {self.filepath}")
        except Exception as e:
            print(f"File transfer failed: {e}")

async def main():
    client = FileTransferClient(
        "alice@example.com",
        "password",
        "bob@example.com",
        "/path/to/document.pdf"
    )
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

Message Archive Management (MAM)

MAM (XEP-0313) allows clients to retrieve message history stored on the server. This is essential for multi-device synchronization and for displaying conversation history when a user comes back online:

import asyncio
import slixmpp
from datetime import datetime

class MAMClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.add_event_handler("session_start", self.start)

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Retrieve message history with a specific contact
        print("=== Message History with bob@example.com ===")
        results = await self['xep_0313'].retrieve(
            with_jid="bob@example.com",
            iterator=True,
            rsm={'max': 50}
        )

        async for page in results:
            for msg in page['mam']['results']:
                if msg['mam_result']['forwarded']['message']['body']:
                    timestamp = msg['mam_result']['forwarded']['delay']['stamp']
                    sender = msg['mam_result']['forwarded']['message']['from']
                    body = msg['mam_result']['forwarded']['message']['body']
                    print(f"[{timestamp}] {sender}: {body}")

async def main():
    client = MAMClient("alice@example.com", "password")
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

Building a Custom Extension

One of XMPP's greatest strengths is its extensibility. You can define custom XML namespaces and stanza extensions for application-specific data. Here is how to create a custom extension in Slixmpp:

import slixmpp
from slixmpp.xmlstream import ElementBase, register_stanza_plugin
from slixmpp.xmlstream.handler import Callback
from slixmpp.xmlstream.matcher import StanzaPath

# Define a custom XML element
class LocationUpdate(ElementBase):
    name = "location"
    namespace = "urn:custom:location"
    plugin_attrib = "location"
    interfaces = {"latitude", "longitude", "description"}

    def get_latitude(self):
        return self._get_sub_text("latitude")

    def set_latitude(self, value):
        return self._set_sub_text("latitude", value)

    def get_longitude(self):
        return self._get_sub_text("longitude")

    def set_longitude(self, value):
        return self._set_sub_text("longitude", value)

    def get_description(self):
        return self._get_sub_text("description")

    def set_description(self, value):
        return self._set_sub_text("description", value)

class LocationClient(slixmpp.ClientXMPP):
    def __init__(self, jid, password):
        slixmpp.ClientXMPP.__init__(self, jid, password)
        self.add_event_handler("session_start", self.start)

        # Register the custom stanza plugin
        register_stanza_plugin(slixmpp.Message, LocationUpdate)

        # Register a handler for incoming location updates
        self.register_handler(
            Callback(
                "Location Update",
                StanzaPath("message/location"),
                self.handle_location
            )
        )

    async def start(self, event):
        self.send_presence()
        await self.get_roster()

        # Send a location update via message
        msg = self.make_message(
            mto="bob@example.com",
            mtype="headline"
        )
        msg['location']['latitude'] = "37.7749"
        msg['location']['longitude'] = "-122.4194"
        msg['location']['description'] = "San Francisco, CA"
        msg.send()
        print("Location update sent!")

    def handle_location(self, msg):
        loc = msg['location']
        print(f"Location from {msg['from'].bare}:")
        print(f"  Lat: {loc['latitude']}")
        print(f"  Lon: {loc['longitude']}")
        print(f"  Desc: {loc['description']}")

async def main():
    client = LocationClient("alice@example.com", "password")
    await client.connect()
    await client.process(forever=False)

asyncio.run(main())

The resulting XML stanza would look like:

<message
    from='alice@example.com/desktop'
    to='bob@example.com'
    type='headline'
    id='loc1'>
  <location xmlns='urn:custom:location'>
    <latitude>37.7749</latitude>
    <longitude>-122.4194</longitude>
    <description>San Francisco, CA</description>
  </location>
</message>

Server-to-Server Communication

Federation is a cornerstone of XMPP. Server-to-server (S2S) communication uses the Server Dialback protocol (XEP-0220) or direct TLS with SASL EXTERNAL authentication. When Alice on example.com sends a message to Bob on example.org, the servers establish a connection:

<!-- example.com initiates S2S stream to example.org -->
<stream:stream
    xmlns='jabber:server'
    xmlns:stream='http://etherx.jabber.org/streams'
    to='example.org'
    from='example.com'
    version='1.0'>

<!-- Server Dialback request -->
<db:result
    from='example.com'
    to='example.org'>
  dGhpcyBpcyBhIGRpYWxiYWNrIGtleQ==
</db:result>

<!-- After verification, messages can flow -->
<message
    from='alice@example.com/desktop'
    to='bob@example.org'
    type='chat'>
  <body>Hello Bob on a different server!</body>
</message>

For developers building server components, here is how to connect a component to an XMPP server using the Jabber Component Protocol (XEP-0114):

import asyncio
import slixmpp.component

class CustomComponent(slixmpp.componentxmpp.ComponentXMPP):
    def __init__(self, jid, secret, server, port):
        slixmpp.componentxmpp.ComponentXMPP.__init__(
            self, jid, secret, server, port
        )
        self.add_event_handler("session_start", self.start)
        self.add_event_handler("message", self.on_message)

    async def start(self, event):
        print(f"Component connected: {self.boundjid}")
        self.send_presence()

    def on_message(self, msg):
        print(f"Component received: {msg['body']} from {msg['from']}")
        
        # Process the message and respond
        response = self.make_message(
            mto=msg['from'],
            mbody=f"Processed: {msg['body']}"
        )
        response.send()

async def main():
    component = CustomComponent(
        "weather.example.com",
        "shared_secret",
        "example.com",
        5347
    )
    await component.connect()
    await component.process(forever=False)

asyncio.run(main())

Best Practices

Security Best Practices

Security should be a primary concern when building XMPP applications. Follow these guidelines: