Real-time Experience at FOSDEM 2018

by Bartłomiej Górny, Michał Piotrowski, Wioletta Dec

This year we took our chance to engage with one of the largest open source software families — by attending FOSDEM. This is a Free Open Source European Developer’s Meeting taking place at the Université Libre de Bruxelles Solbosch campus in Brussels on 3–4 February. FOSDEM is among the world’s largest events focusing on free and open-source software development and its atmosphere is pretty unique.

Fig. 1: ‘Elasticsearch (R)Evolution’, presented by Philipp Krenn, Fosdem 2018

FOSDEM 2018 was incredible as always — huge, noisy, crowded, crazy, brimming with energy, joy and never ending excitement. With nearly 14,000 attendees, 653 speakers and almost 700 talks supporting open source innovations, spread across 57 thematic sessions, the event was a hit. Most of the conference rooms were full to bursting, the exhibitors and networking areas were busy from early morning until evening and filled with enormous number of free open source projects and their communities. Github-sponsored free coffee, stalls featuring everything from hardcore Perl hackers to the VLC squad dressed as roadwork cones and LizardFS’ Polish-Irish team treating visitors to their stand with vodka shot!

Fig. 2: Exhibitors Area, FOSDEM 2018

Alessio Fattorini described FOSDEM as “Woodstock for Geeks”. Taking the level of enthusiasm, creative energy combined with tech, innovation and knowledge, we’d say he was right!

As always, organisers did a great job increasing networking and knowledge sharing opportunities through multiple accompanying social events. 2018 is the 20th anniversary of Open Source and it was celebrated at Fosdem.

We were there too, representing our own open source based messaging platform MongooseIM, while hanging out in the Real-Time Lounge.

REAL-TIME LOUNGE

The ‘Real-Time Lounge’ is where various open source, XMPP powered software solutions are presented by their communities, under the umbrella XSF (XMPP Standards Foundation) who are the main hosts of the ‘Real Time Lounge’.

Squeezed between vending machines and a wheel of fortune was our cosy corner, where we brought together the best of open-source instant messaging and real-time communication. XSF, first and foremost, and XMPP-based software — Tigase, Prosody, MongooseIM, OpenFire and more.

Fig. 3: Real Time Lounge, FOSDEM 2018

In addition, our MongooseIM representatives, Michał Piotrowski and Bartłomiej Górny presented their work within the Real Time Communications Devroom. While Michał explored different ways of scaling an open source XMPP server — MongooseIM, Bartłomiej shared his vision of XMPP as the road to innovation.

Michał Piotrowski, “Scaling messaging systems”

“In my talk I described the most frequent scalability limitations when it comes to MongooseIM, showing that, thanks to the Erlang VM, scaling a single node MongooseIM installation or multiple nodes in the same datacenter is easy. The real fun begins when we need to spread the installation between continents. In this case there are two options. XMPP federation where users are bound to a specific datacenter. There is also the new MongooseIM feature called Geo Distribution — in this cases all MongooseIM clusters across the globe serve the same XMPP domain, so users can connect to any datacenter, usually the closest one.”

Scalability depends on many variables for MongooseIM’s or indeed any other XMPP server. It’s hard to say that a particular option will work for every one, as every MongooseIM installation we have had the pleasure to build is unique. When planning how to scale MongooseIM we need to consider things like:

1. Machine power (CPU and memory)
2. Type of connected users. Using an XMPP server from a web browser or a desktop is different from a mobile device. A mobile device will (re-)connect more often than a desktop client.
3. Used and enabled XMPP features. Using only basic one-to-one messages without archiving will place much lighter demands on the server than for instance large group chats with many active users.

When the service popularity grows and there are more and more users generating more and more load on the server, resources may be fully utilised. Typically it’s the memory which runs out first, but depending on your specific use case it may happen that first you will run out of CPU power or maybe your database will start slowing down.

When you hit one of the limits, you need to start scaling MongooseIM. There is several ways to do it.

First of all, thanks to Erlang VM, scaling a single MongooseIM node is as easy as adding more resources (more RAM or more powerful CPU). Every connected device is represented as 2–3 lightweight Erlang processes. These processes are then spread across all available CPU cores thanks to Erlang scheduler.

A single installation, while able to handle millions of users on very powerful machines, is usually a bad idea due to having a single point of failure. Typical production installation consists of at least 3 MongooseIM nodes to be able to survive single node crashes. With multi-node installation we need to remember to keep enough spare resources on all the nodes to be able to handle traffic from a dead node. At some point there may be a need to add more nodes to cluster. This again, thanks to the Erlang VM, is very easy. MongooseIM uses the Erlang distribution layer and Mnesia database to set up the cluster and communicate between nodes in the cluster.

In some cases one cluster is not enough. In this case we can scale MongooseIM beyond a cluster. The simplest solution is XMPP federation. In this approach we setup several MongooseIM clusters, and every cluster serves one XMPP domain. With this setup, users will be bound to a specific domain and will always need to be connected to the same datacenter. Sometimes this is too limiting and we’d like to allow our users to connect to the closest datacenter. And there is answer to this need — it’s MongooseIM’s new extension called Geo Distribution. Thanks to this, many different MongooseIM clusters can serve the same domain and users can connect to any (usually the closest) datacenter.

Watch video recording here.

Bartłomiej Górny, “XMPP as the road to innovation”

“I think the extensibility of XMPP is the best of both worlds: it has a built-in mechanism that makes it really easy to extend, to build stuff on top of it, but it also has the XSF which has a firm grip on it, making sure it grows in a controlled, orderly way. The protocol, with all its official extensions, covers a wide scope of functionalities, but still is clean and tidy. This is what makes it such a good tool for innovative implementations.”

XMPP is designed to be extensible, to serve as a vehicle for innovation. For this purpose, it provides technical means for building solutions on top of the existing protocol. But extensibility alone is nowhere near enough to be truly useful — for example, Drupal, a CMS written in PHP, features over 20k extensions. Bad news for someone trying to find something he needs.

To be a really good platform for innovation, XMPP provides three pillars:

1. A solid, stable foundation
2. Built-in extension mechanism
3. Procedures to verify and share extensions

1. Foundation

Basically, XMPP is for exchanging messages and presence information among clients. It is a client-server protocol, based on XML. Core XMPP documents (RFCs) define the addressing scheme, setting up TCP session and procedures for presence management and exchange and basic messaging.

The base “layer” of the protocol are three XML structures (called “stanzas”), each categorised in a few types:

1. IQ (get, set, result, error)
2. Presence (available, unavailable, probe, plus a few more specialised types)
3. Message (normal, chat, groupchat, …)

Fig. 4: A simple message stanza

Each stanza has a defined role: IQ (info/query) is basically for communicating with server; presence conveys information about the client’s availability, both to server and to the client’s friends; message is client-client communication and can server a wide variety of purposes.

These stanzas are the core of the protocol, and are not meant to be changed. The rest is extensions, documented in “XMPP Enhancement Proposals” (XEPs). There are quite a few of them, but not too many — just a few hundred, which may still sound scary, but it is just because there is a lot of stuff you can do with XMPP. And here comes the best part: there is a list of officially supported XEPs, maintained by XSF (XMPP Standards Foundation), available on xmpp.org. Everything on this list has been thoroughly scrutinised, discussed, proofread, improved and voted on by people from the foundation. In other words, you can trust them and build your own implementation on top of. It is also easy to choose the one you need, because they usually do not overlap — if you find two XEPs cover similar functionalities, then most likely one of them is already marked as deprecated, or soon will be.

Fig. 5: Part of the list of XMPP extensions

2. Extension mechanism

While the base layer of the protocol is defined by a narrow selection of xml tag names and attributes (message/chat etc), extensions are built around the concept of namespaces. The whole mechanism can be described in just a few bullet points:

1. an extension uses a stanza as its transport (message, presence or iq)
2. an extension defines its unique namespace
3. an extension is documented in its own XEP Therefore there is a one-to-one-to-one mapping between extensions, namespaces and XEPs.

Fig. 6: Namespace identifying protocol extensions

This is the gist of it — namespace uniquely identifies the feature being used, is human-readable, and makes it easy to plug in a dedicated handler for a given extension.

3. Verification

If you design your own extension, the best way to test it is to submit it to XSF. The Foundation is always looking for new ideas, but is also extremely careful and demanding. If you submit your document (the process is described on https://xmpp.org/about/standards-process.html) it will be really thoroughly evaluated, and you will have to spend lots of time answering questions and making corrections and improvement. Chances it makes it to the official list are rather slim, but you will have your feedback galore. The whole process is time consuming and often frustrating, it also rather taxing from the Foundation’s point of view. It is, though, the only way to make sure the protocol is not getting polluted with half-baked ideas.

The right approach to extending

If you think about extending XMPP, here is a list of steps to take:

Step 1: Read carefully a list of XEPs on xmpp.org. Not just once.

As stated before, there are quite a few of them, covering many areas, and the chances are what you need is already taken care of. It might not be obvious at the first glance, so a close look is highly recommended.

Step 2: Read the list again to see what can be used as the base

An extension is not a fixed take-it-or-leave it thing. It is more like a class in OOP, something you can use as-is or build another class on top of it. Remember: namespaces can be nested, so there is nothing stopping you from implementing extension within extension.

Fig. 7: Nested namespaces — extending an extension

Step 3: Choose stanza carefully

If there is nothing that would suit your purpose and you have to design an extension from scratch, think twice before choosing a stanza. If it is for client-server communication (like getting some data from server, setting parameters, performing administrative action) most likely you will go with IQ. For nearly all other purposes use message. Yes, it is tempting to use presence, sometimes it even seems obvious that it should be a presence. Let me give you an example, then: suppose we want to publish a user’s geographical location in real time. The first idea is “let’s use presence”. It is quite logical, since the user is “present” at a certain location. Let’s design an extension which would add a geolocation element to a presence, and voila, we’re done.

Fig. 8: The wrong way to propagate user state info

It turns out that it is a very bad idea. First, presence goes out to everybody in your roster (contact list), whether he wants it or not. And he can’t get rid of it, unless he unsubscribes, which means he is not going to receive any presence at all, location or no location. Plus, presence info is never archived, which is something you might want. If you must, use message.

By the way, if you consider it at all, it means you haven’t executed step 2 — there is XEP-0163 “Personal Eventing Protocol” which is a perfect tool for the job. It turns every user account to a pub-sub node to which you can subscribe and receive event information, location in this case, as messages. Even worse, it means you haven’t even executed step 1, because if you had, you’d have had discovered XEP-0080 which does exactly what you need.

Step 4 — write a XEP

It is highly recommend to document your extension properly. Of course, it is always highly recommended to properly document anything you do; my advice here is to document it in the same way all other extensions are documented. It is quite verbose, but at least it is consistent; programmers implementing your idea will have to read lots of XEPs AND the one you are writing. Also, consider submitting it to XSF. Even if it is rejected (which it probably will), you will get lots of useful feedback, free of charge.

Watch video recording here.

As this year we are celebrating the 20th anniversary of open-sourced Erlang #OpenErlang, we’ve been glad to be part of FOSDEM, connect with a global open source community and celebrate 20 years of open sourced Erlang together.

Real-time Experience at FOSDEM 2018 was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.