Introduction to XMPP
XMPP, which stands for Extensible Messaging and Presence Protocol, is an open XML-based communication protocol designed for real-time messaging, presence information, and contact list maintenance. Originally developed in 1999 by Jeremie Miller as the Jabber protocol, XMPP has evolved into a mature, decentralized, and highly extensible standard maintained by the XMPP Standards Foundation (XSF).
What is XMPP?
At its core, XMPP is a protocol for streaming XML elements between network endpoints. Unlike many modern messaging systems that rely on centralized servers controlled by a single company, XMPP uses a federated architecture similar to email. Anyone can run an XMPP server, and servers can communicate with each other, allowing users on different servers to exchange messages seamlessly.
The protocol operates over TCP connections and uses XML streams to exchange structured data. Each connection establishes a long-lived XML stream from client to server, and another stream from server to client. Within these streams, individual XML stanzas are sent back and forth to carry out operations like authentication, messaging, and presence updates.
Why XMPP Matters
XMPP remains relevant in modern software development for several compelling reasons:
- Open Standard: The protocol is fully documented, open, and free from proprietary lock-in. Anyone can implement it without licensing fees.
- Federation: Like email, XMPP servers can federate, meaning users on different domains can communicate without being on the same platform.
- Extensibility: XMPP uses a modular extension system called XEPs (XMPP Extension Protocols), allowing developers to add features without modifying the core protocol.
- Proven at Scale: XMPP powers major platforms including WhatsApp's internal messaging infrastructure, Google's historical chat services, and numerous enterprise communication systems.
- Security: Built-in support for TLS encryption and SASL authentication makes XMPP suitable for secure communications.
- Real-Time Capability: The persistent connection model enables instant message delivery and presence updates without polling.
XMPP Architecture
Understanding the XMPP architecture is essential before building any application on top of it. The protocol defines a client-server model with federation between servers.
Core Components
The XMPP ecosystem consists of several key components that work together:
- Clients: Applications that connect to an XMPP server to send and receive messages. Clients authenticate with a server and maintain a persistent XML stream.
- Servers: Central hubs that route messages, manage user accounts, handle presence, and federate with other servers. Popular server implementations include Prosody, ejabberd, Openfire, and Tigase.
- Gateways/Transports: Components that bridge XMPP with other protocols like IRC, SMS, or legacy instant messaging systems.
- Components: Server-side add-ons that provide additional services like Multi-User Chat (MUC), PubSub, or file transfer proxies.
Addressing Scheme
Every entity in XMPP is identified by a Jabber ID (JID), which follows a format similar to an email address. Understanding JID structure is fundamental:
user@domain.tld/resource
The JID has three parts:
- localpart (user): The username, which is unique within a server domain.
- domainpart (domain.tld): The server hostname that hosts the user's account.
- resourcepart (resource): An optional identifier that distinguishes multiple simultaneous connections from the same user (e.g., "phone", "desktop", "web").
A JID without a resource part is called a "bare JID" and identifies a user account. A JID with a resource part is a "full JID" and identifies a specific connected session. This distinction matters when routing messages, as sending to a bare JID lets the server decide which session receives the message, while sending to a full JID targets a specific session.
Getting Started with XMPP
To begin developing with XMPP, you need access to an XMPP server and a client library for your programming language of choice.
Setting up a Server
For development purposes, Prosody is an excellent choice due to its simplicity and Lua-based configuration. Here is a basic Prosody configuration file:
-- /etc/prosody/prosody.cfg.lua
-- Server-wide settings
admins = { "admin@example.com" }
-- Enable modules
modules_enabled = {
"roster"; -- Contact list management
"saslauth"; -- Authentication
"tls"; -- Encryption
"dialback"; -- Server-to-server authentication
"disco"; -- Service discovery
"ping"; -- Keep-alive
"pep"; -- Personal eventing
"carbons"; -- Message carbons
"mam"; -- Message archive management
"muc"; -- Multi-user chat
"pubsub"; -- Publish-subscribe
}
-- Virtual hosts
VirtualHost "example.com"
enabled = true
ssl = {
key = "/etc/prosody/certs/example.com.key";
certificate = "/etc/prosody/certs/example.com.crt";
}
-- Component for group chat
Component "conference.example.com" "muc"
-- Component for PubSub
Component "pubsub.example.com" "pubsub"
After saving the configuration, restart Prosody and create a test user:
prosodyctl adduser alice@example.com
prosodyctl adduser bob@example.com
Client Libraries
XMPP client libraries exist for virtually every programming language. Some popular options include:
- Python: Slixmpp, aioxmpp, sleekxmpp
- JavaScript/Node.js: @xmpp/client, stanza.io, node-xmpp
- Java: Smack
- Go: mellium.im/xmpp
- Rust: xmpp-rs, tokio-xmpp
- C/C++: libstrophe, gloox
Connecting and Authenticating
The first step in any XMPP application is establishing a connection and authenticating. The connection process involves opening an XML stream, negotiating TLS, and authenticating via SASL. Here is a Python example using Slixmpp:
import asyncio
import slixmpp
from slixmpp.xmlstream import ET
class XMPPClient(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.session_start)
self.add_event_handler("message", self.message_handler)
self.add_event_handler("changed_status", self.presence_handler)
async def session_start(self, event):
# Send initial presence
self.send_presence()
# Request the roster (contact list)
await self.get_roster()
print("Session started and presence sent!")
def message_handler(self, msg):
if msg['type'] in ('chat', 'normal'):
print(f"Message from {msg['from']}: {msg['body']}")
def presence_handler(self, presence):
print(f"Presence update: {presence['from']} is {presence['type'] or 'available'}")
async def main():
client = XMPPClient("alice@example.com", "password")
await client.connect()
await client.process(forever=False)
asyncio.run(main())
For JavaScript developers, the @xmpp/client package provides a modern, promise-based API:
const { client, xml } = require("@xmpp/client");
const debug = require("@xmpp/debug");
const xmpp = client({
service: "xmpp://example.com:5222",
domain: "example.com",
username: "alice",
password: "password",
});
debug(xmpp, true);
xmpp.on("error", (err) => {
console.error("XMPP error:", err);
});
xmpp.on("offline", () => {
console.log("Client is offline");
});
xmpp.on("stanza", async (stanza) => {
console.log("Received stanza:", stanza.toString());
});
xmpp.on("online", async (address) => {
console.log("Connected as", address.toString());
// Send initial presence
await xmpp.send(xml("presence"));
// Send a message
const message = xml(
"message",
{ type: "chat", to: "bob@example.com" },
xml("body", {}, "Hello Bob from Node.js!")
);
await xmpp.send(message);
});
xmpp.start().catch(console.error);
Under the hood, the connection process follows this XML exchange. The client opens a stream:
<?xml version='1.0'?>
<stream:stream
xmlns='jabber:client'
xmlns:stream='http://etherx.jabber.org/streams'
to='example.com'
version='1.0'>
The server responds with its stream and available authentication mechanisms:
<stream:stream
xmlns='jabber:client'
xmlns:stream='http://etherx.jabber.org/streams'
from='example.com'
id='abc123'
version='1.0'>
<stream:features>
<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'>
<required/>
</starttls>
<mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>
<mechanism>SCRAM-SHA-256</mechanism>
<mechanism>PLAIN</mechanism>
</mechanisms>
</stream:features>
The Three Core Stanza Types
XMPP defines three fundamental stanza types that form the basis of all communication. Understanding these stanzas is critical for any XMPP developer.
Message Stanzas
Message stanzas are used for one-to-one communication, group chat, and notifications. They are "fire and forget" by default, meaning the server does not confirm delivery unless explicitly requested:
<message
from='alice@example.com/desktop'
to='bob@example.com'
type='chat'
id='msg001'>
<body>Hello Bob, how are you?</body>
<active xmlns='http://jabber.org/protocol/chatstates'/>
</message>
The type attribute on a message stanza determines its context:
- chat: A one-to-one instant message between two users.
- groupchat: A message sent to a multi-user chat room.
- normal: A standalone message that does not expect an immediate reply.
- headline: A notification or alert that typically should not be stored offline.
- error: An error response to a previously sent message.
Presence Stanzas
Presence stanzas communicate availability status. They are the mechanism that makes XMPP a "presence protocol" rather than just a messaging protocol:
<presence from='alice@example.com/desktop'>
<show>away</show>
<status>In a meeting</status>
<priority>10</priority>
</presence>
The show element can contain one of four values: away, chat (available for chat), dnd (do not disturb), or xa (extended away). The status element holds a human-readable status message, and priority determines which resource receives messages sent to the bare JID.
IQ Stanzas
IQ (Info/Query) stanzas implement a request-response pattern, similar to HTTP. They always require an id attribute for matching responses and use a type attribute of get, set, result, or error:
<!-- Request: Get Bob's version info -->
<iq from='alice@example.com/desktop'
to='bob@example.com/laptop'
type='get'
id='version1'>
<query xmlns='jabber:iq:version'/>
</iq>
<!-- Response -->
<iq from='bob@example.com/laptop'
to='alice@example.com/desktop'
type='result'
id='version1'>
<query xmlns='jabber:iq:version'>
<name>MyCustomClient</name>
<version>1.0.0</version>
<os>Linux 5.15.0</os>
</query>
</iq>
Working with the Roster
The roster is XMPP's contact list system. Managing the roster involves IQ stanzas for retrieving, adding, and removing contacts, combined with presence subscriptions for seeing contacts' availability.
import slixmpp
from slixmpp.exceptions import IqError, IqTimeout
class RosterManager(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.start)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Add a new contact
await self.add_contact("charlie@example.com", "Charlie from work")
# Retrieve and display the roster
roster = self.client_roster
for jid in roster:
print(f"Contact: {jid}")
print(f" Name: {roster[jid]['name']}")
print(f" Subscription: {roster[jid]['subscription']}")
print(f" Groups: {roster[jid]['groups']}")
async def add_contact(self, jid, name=None):
"""Add a contact to the roster and send a subscription request."""
self.send_presence(pto=jid, ptype="subscribe")
self.update_roster(jid, name=name)
print(f"Subscription request sent to {jid}")
async def remove_contact(self, jid):
"""Remove a contact from the roster."""
self.send_presence(pto=jid, ptype="unsubscribe")
self.send_presence(pto=jid, ptype="unsubscribed")
await self.del_roster_item(jid)
print(f"Removed {jid} from roster")
The subscription model in XMPP is bidirectional. When Alice wants to see Bob's presence, she sends a subscribe request. Bob must approve it with a subscribed response. If Bob also wants to see Alice's presence, he sends his own subscribe request. The subscription states are:
- none: No subscription in either direction.
- to: The user is subscribed to the contact's presence.
- from: The contact is subscribed to the user's presence.
- both: Both parties are subscribed to each other's presence.
Messaging
One-to-One Chat
Sending a direct message is straightforward. Here is a complete example that sends a message and handles replies:
import asyncio
import slixmpp
from slixmpp.xmlstream import ET
class ChatClient(slixmpp.ClientXMPP):
def __init__(self, jid, password, recipient, message):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.recipient = recipient
self.message = message
self.add_event_handler("session_start", self.start)
self.add_event_handler("message", self.on_message)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Send the message
self.send_message(
mto=self.recipient,
mbody=self.message,
mtype='chat'
)
print(f"Message sent to {self.recipient}")
def on_message(self, msg):
if msg['type'] in ('chat', 'normal'):
print(f"\nReply from {msg['from'].bare}:")
print(f" {msg['body']}")
# Handle chat state notifications
if msg['chat_state']:
print(f" [Chat state: {msg['chat_state']}]")
async def main():
client = ChatClient(
"alice@example.com",
"password",
"bob@example.com",
"Hey Bob! This is a test message."
)
await client.connect()
await client.process(forever=False)
asyncio.run(main())
Group Chat (MUC)
Multi-User Chat (MUC), defined in XEP-0045, enables group chat rooms. Joining and participating in a MUC room requires specific stanza handling:
import asyncio
import slixmpp
from slixmpp.exceptions import MucError
class MUCClient(slixmpp.ClientXMPP):
def __init__(self, jid, password, room, nick):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.room = room
self.nick = nick
self.add_event_handler("session_start", self.start)
self.add_event_handler("groupchat_message", self.on_muc_message)
self.add_event_handler("muc::%s::got_online" % room, self.on_user_join)
self.add_event_handler("muc::%s::got_offline" % room, self.on_user_leave)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Join the MUC room
await self.plugin['xep_0045'].join_muc(
self.room,
self.nick,
maxhistory="20"
)
print(f"Joined room: {self.room} as {self.nick}")
def on_muc_message(self, msg):
# Ignore messages from ourselves
if msg['mucnick'] == self.nick:
return
print(f"[{msg['from'].resource}] {msg['body']}")
def on_user_join(self, presence):
print(f"*** {presence['muc']['nick']} joined the room")
def on_user_leave(self, presence):
print(f"*** {presence['muc']['nick']} left the room")
async def send_to_room(self, message):
self.send_message(
mto=self.room,
mbody=message,
mtype='groupchat'
)
async def main():
client = MUCClient(
"alice@example.com",
"password",
"devteam@conference.example.com",
"AliceDev"
)
await client.connect()
# Send a message after joining
await asyncio.sleep(2)
await client.send_to_room("Hello everyone in the dev team!")
await client.process(forever=False)
asyncio.run(main())
The raw XML for joining a MUC room looks like this:
<presence
from='alice@example.com/desktop'
to='devteam@conference.example.com/AliceDev'>
<x xmlns='http://jabber.org/protocol/muc'>
<history maxchars='0'/>
</x>
</presence>
Service Discovery
Service Discovery (XEP-0030) allows clients and components to discover what features and services an entity supports. This is how XMPP achieves its extensibility — clients can dynamically detect server capabilities:
import asyncio
import slixmpp
class DiscoveryClient(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.start)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Discover server features
print("=== Server Features ===")
try:
info = await self['xep_0030'].get_info(jid="example.com")
for feature in info['disco_info']['features']:
print(f" Feature: {feature}")
for identity in info['disco_info']['identities']:
print(f" Identity: {identity}")
except Exception as e:
print(f"Discovery error: {e}")
# Discover available services (MUC, PubSub, etc.)
print("\n=== Server Items ===")
try:
items = await self['xep_0030'].get_items(jid="example.com")
for item in items['disco_items']['items']:
print(f" Service: {item[0]}")
# Discover features of each service
sub_info = await self['xep_0030'].get_info(jid=item[0])
for identity in sub_info['disco_info']['identities']:
print(f" Type: {identity['type']}, Name: {identity['name']}")
except Exception as e:
print(f"Items error: {e}")
async def main():
client = DiscoveryClient("alice@example.com", "password")
await client.connect()
await client.process(forever=False)
asyncio.run(main())
The underlying IQ stanza for service discovery is:
<iq from='alice@example.com/desktop'
to='example.com'
type='get'
id='disco1'>
<query xmlns='http://jabber.org/protocol/disco#info'/>
</iq>
Publish-Subscribe (PubSub)
XMPP PubSub (XEP-0060) provides a publish-subscribe pattern that enables many advanced features like news feeds, avatar distribution, and event notifications. A node is created on the PubSub service, publishers push items to it, and subscribers receive notifications:
import asyncio
import slixmpp
from slixmpp.xmlstream import ET
class PubSubClient(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.pubsub_server = "pubsub.example.com"
self.add_event_handler("session_start", self.start)
async def start(self, event):
self.send_presence()
await self.get_roster()
node_name = "urn:example:notifications"
# Create a node
try:
await self['xep_0060'].create_node(
self.pubsub_server,
node_name
)
print(f"Created node: {node_name}")
except Exception as e:
print(f"Node may already exist: {e}")
# Subscribe to the node
try:
await self['xep_0060'].subscribe(
self.pubsub_server,
node_name
)
print(f"Subscribed to node: {node_name}")
except Exception as e:
print(f"Subscription error: {e}")
# Publish an item
payload = ET.Element("{urn:example:notifications}event")
title = ET.SubElement(payload, "title")
title.text = "System Update"
desc = ET.SubElement(payload, "description")
desc.text = "Maintenance scheduled for tonight at 2 AM UTC."
try:
result = await self['xep_0060'].publish(
self.pubsub_server,
node_name,
payload=payload,
id="event-001"
)
print(f"Published item: {result}")
except Exception as e:
print(f"Publish error: {e}")
# Retrieve published items
try:
items = await self['xep_0060'].get_items(
self.pubsub_server,
node_name,
max_items=10
)
print(f"\nRetrieved items:")
for item in items['pubsub']['items']:
print(f" Item ID: {item['id']}")
except Exception as e:
print(f"Retrieve error: {e}")
async def main():
client = PubSubClient("alice@example.com", "password")
await client.connect()
await client.process(forever=False)
asyncio.run(main())
File Transfer
File transfer in XMPP is handled through XEP-0096 (SI File Transfer) combined with XEP-0065 (SOCKS5 Bytestreams) or XEP-0047 (In-Band Bytestreams). Here is an example using In-Band Bytestreams, which works through the existing XMPP stream:
import asyncio
import slixmpp
class FileTransferClient(slixmpp.ClientXMPP):
def __init__(self, jid, password, recipient, filepath):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.recipient = recipient
self.filepath = filepath
self.add_event_handler("session_start", self.start)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Send a file using In-Band Bytestreams
try:
await self['xep_0096'].send_file(
self.recipient,
self.filepath,
description="Project documentation"
)
print(f"File sent: {self.filepath}")
except Exception as e:
print(f"File transfer failed: {e}")
async def main():
client = FileTransferClient(
"alice@example.com",
"password",
"bob@example.com",
"/path/to/document.pdf"
)
await client.connect()
await client.process(forever=False)
asyncio.run(main())
Message Archive Management (MAM)
MAM (XEP-0313) allows clients to retrieve message history stored on the server. This is essential for multi-device synchronization and for displaying conversation history when a user comes back online:
import asyncio
import slixmpp
from datetime import datetime
class MAMClient(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.start)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Retrieve message history with a specific contact
print("=== Message History with bob@example.com ===")
results = await self['xep_0313'].retrieve(
with_jid="bob@example.com",
iterator=True,
rsm={'max': 50}
)
async for page in results:
for msg in page['mam']['results']:
if msg['mam_result']['forwarded']['message']['body']:
timestamp = msg['mam_result']['forwarded']['delay']['stamp']
sender = msg['mam_result']['forwarded']['message']['from']
body = msg['mam_result']['forwarded']['message']['body']
print(f"[{timestamp}] {sender}: {body}")
async def main():
client = MAMClient("alice@example.com", "password")
await client.connect()
await client.process(forever=False)
asyncio.run(main())
Building a Custom Extension
One of XMPP's greatest strengths is its extensibility. You can define custom XML namespaces and stanza extensions for application-specific data. Here is how to create a custom extension in Slixmpp:
import slixmpp
from slixmpp.xmlstream import ElementBase, register_stanza_plugin
from slixmpp.xmlstream.handler import Callback
from slixmpp.xmlstream.matcher import StanzaPath
# Define a custom XML element
class LocationUpdate(ElementBase):
name = "location"
namespace = "urn:custom:location"
plugin_attrib = "location"
interfaces = {"latitude", "longitude", "description"}
def get_latitude(self):
return self._get_sub_text("latitude")
def set_latitude(self, value):
return self._set_sub_text("latitude", value)
def get_longitude(self):
return self._get_sub_text("longitude")
def set_longitude(self, value):
return self._set_sub_text("longitude", value)
def get_description(self):
return self._get_sub_text("description")
def set_description(self, value):
return self._set_sub_text("description", value)
class LocationClient(slixmpp.ClientXMPP):
def __init__(self, jid, password):
slixmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler("session_start", self.start)
# Register the custom stanza plugin
register_stanza_plugin(slixmpp.Message, LocationUpdate)
# Register a handler for incoming location updates
self.register_handler(
Callback(
"Location Update",
StanzaPath("message/location"),
self.handle_location
)
)
async def start(self, event):
self.send_presence()
await self.get_roster()
# Send a location update via message
msg = self.make_message(
mto="bob@example.com",
mtype="headline"
)
msg['location']['latitude'] = "37.7749"
msg['location']['longitude'] = "-122.4194"
msg['location']['description'] = "San Francisco, CA"
msg.send()
print("Location update sent!")
def handle_location(self, msg):
loc = msg['location']
print(f"Location from {msg['from'].bare}:")
print(f" Lat: {loc['latitude']}")
print(f" Lon: {loc['longitude']}")
print(f" Desc: {loc['description']}")
async def main():
client = LocationClient("alice@example.com", "password")
await client.connect()
await client.process(forever=False)
asyncio.run(main())
The resulting XML stanza would look like:
<message
from='alice@example.com/desktop'
to='bob@example.com'
type='headline'
id='loc1'>
<location xmlns='urn:custom:location'>
<latitude>37.7749</latitude>
<longitude>-122.4194</longitude>
<description>San Francisco, CA</description>
</location>
</message>
Server-to-Server Communication
Federation is a cornerstone of XMPP. Server-to-server (S2S) communication uses the Server Dialback protocol (XEP-0220) or direct TLS with SASL EXTERNAL authentication. When Alice on example.com sends a message to Bob on example.org, the servers establish a connection:
<!-- example.com initiates S2S stream to example.org -->
<stream:stream
xmlns='jabber:server'
xmlns:stream='http://etherx.jabber.org/streams'
to='example.org'
from='example.com'
version='1.0'>
<!-- Server Dialback request -->
<db:result
from='example.com'
to='example.org'>
dGhpcyBpcyBhIGRpYWxiYWNrIGtleQ==
</db:result>
<!-- After verification, messages can flow -->
<message
from='alice@example.com/desktop'
to='bob@example.org'
type='chat'>
<body>Hello Bob on a different server!</body>
</message>
For developers building server components, here is how to connect a component to an XMPP server using the Jabber Component Protocol (XEP-0114):
import asyncio
import slixmpp.component
class CustomComponent(slixmpp.componentxmpp.ComponentXMPP):
def __init__(self, jid, secret, server, port):
slixmpp.componentxmpp.ComponentXMPP.__init__(
self, jid, secret, server, port
)
self.add_event_handler("session_start", self.start)
self.add_event_handler("message", self.on_message)
async def start(self, event):
print(f"Component connected: {self.boundjid}")
self.send_presence()
def on_message(self, msg):
print(f"Component received: {msg['body']} from {msg['from']}")
# Process the message and respond
response = self.make_message(
mto=msg['from'],
mbody=f"Processed: {msg['body']}"
)
response.send()
async def main():
component = CustomComponent(
"weather.example.com",
"shared_secret",
"example.com",
5347
)
await component.connect()
await component.process(forever=False)
asyncio.run(main())
Best Practices
Security Best Practices
Security should be a primary concern when building XMPP applications. Follow these guidelines:
- Always use TLS: Never transmit data over unencrypted connections. Require TLS on both client-to-server and server-to-server connections.
- Use strong SASL mechanisms: Prefer SCRAM-SHA-256 or SCRAM-SHA-1-PLUS over PLAIN authentication. Channel binding (the PLUS variants) prevents man-in-the-middle attacks.
- Validate certificates: Implement proper certificate validation. Do not disable certificate checking in production, even for testing convenience.
- Implement proper error handling: Do not expose internal error details to end users. Use XMPP error stanzas with appropriate error codes.
— Ad —
Google AdSense will appear here after approval