This blog post is about the new design of my wayland libs. It's not a tutorial, but rather about the design choices I made, and why I made them.

I'll start with a quick recap of what wayland is, from an API & protocol point of view. If you already know that, you can directly jump to second section.

Then I'll list the constraints I have to work with regarding my API design. After that, I'll do a quick recap of my two previous design attempts, and then dig into my new design.

It'll be a long article, but I'll try to keep it well structured, so feel free to fast-forward to the parts that interest you the most, if you feel like it's TL;DR. 😉

What's Wayland (the protocol)?

I won't go into details about what Wayland is for, wikipedia does a good job already and this is not the focus of this post. What interests me here is how it works.

Wayland is a message-based and object-oriented protocol: the clients and the compositor (server) exchange messages on an unix socket, these messages being structured on instances of interfaces. Both ends can create objects in a shared namespace, an each message is associated with an object.

Each interface (class) defines a number of requests (messages from the client to the server) and events (messages from the server to the client). Messages can create objects, and they are actually the only way to create object. For example, an instance of wl_compositor will allow the client to create as many wl_surfaces as they want, with the create_surface request.

There are special kind of objects called "global objects". The client can instanciate them at will from an even-more-special object: the wl_registry, which is the entry point of the protocols. The registry will list to the client the global objects that the server have defined, and the client is free to instanciate them as many time as they want and send requests (and receive events).

Note I've used a plural: protocols. While the core wayland protocol defines a number of globals, it is possible to anybody to use a protocol extension, by writing an XML file specifying it. As long as the client and server have both been built using this XML file, they can use the extension. This is determined in the initialisation, as the entry point of this protocol extension will be one more globals. The registry doesn't have these globals? That means the server does not support this extension. Simple.

The design constraints

Designing the API for Wayland on Rust was not an easy task, for several reasons:

Need to use the C libs

The socket protocol being specified, my first idea was to go full-rust, reimplement the whole protocol, and be as free as I want. But...

Mesa is built around the official C libraries. That means that if I want OpenGL and Vulkan to be usable, I must provide pointers that can be passed to mesa, that it'll use with the C library. So I must use the C library (or replicate exactly it's internal structure, but that'd be silly).

So I have to build around the C libraries, and the design choice they made. As unrusty as they can be:

• pointers everywhere, meaning I'll need to build a safe layer around this
• callback-based, meaning I risk lifetime nightmare if I'm not carefull enough (and I experienced it with my first design attempt)
• thread-safe, it's a good thing, but will raise a lot of conccurency questions

Need for code-generation

The protocols are defined in XML files. The C libs come with a tool called wayland-scanner, which uses these files to generate code. The actual ABI of the libs is very small, and only contains message serialization primitives for the socket. All the rest is built by the protocol XML specification.

I'll obviously need to do the same, so I can easily handle upgrades of the xml spec, as well as protocol extensions.

I had this constraint:

• Allow the user to use protocol extensions without needing to patch my lib (my second attempt didn't allow that)

Exploit client-server symmetry

This is not a hard requirement a priori, but more a personal goal. There is a lot of symmetry in the protocol regarding client and server side. I can surely make good use of that symmetry in the code generation, as long as I find a design that can handle the few asymmetries:

• Most objects are created and destroyed by the client, meaning the server will most of the time have to handle objects without controlling their lifespan. This is somehow the absolute opposite of Rust's ownership story.

Previous attempts

1^st try: closures for callbacks

My first try (versions 0.1 and 0.2 of the libs) was very naive: hand-write the whole protocol, with a method for each event where the user provides a closure as a callback.

I quickly understood this could not work, because far too painfull to use: to meet the thread-safety rules, I quickly had to put Send, Sync and 'static bounds in a lot of places, making the whole thing impratical as long as the callbacks needed to interact with a shared state, which is quite common in this context. Had to put Arc<Mutex<_>> everywhere, even in single-threaded applications.

Not a good design at all.

2^nd try: event iterators

For a second try (versions 0.3 to 0.6) I chose to use a nice corner of the wayland API: there is a mechanism allowing to plug callbacks at a deeper level of the message dispaching, and bypass the whole callback hell by instead providing a single callback to get the raw messages and dispatch them manually.

I decided to use this mechanism to replace the callbacks by an iterator of events, which is much more Rust-friendly, but had a few drawbacks: it's no longer possible to provide references to the objects directly as args of the callback, so I had to use opaque IDs. Once receiving an event, the user had to check its id an compare it to their object store to figure out which object is concerned by the event. It was not very good, but it worked.

Client-side only.

This approach was absolutely not possible to use for server-side, because of the lifetime issues. Given most objects are created and destroyed by the client, the server must follow things precisely. Especially, when the client destroys an object in a request, the server must destroy its handle of this object. Failure to do so will completely mess up the internal state of the C libraries and cause the rest of the message exchange to be completely corrupted.

Other issue, the event iterator return type is a large enum, that must be aware of all possible events. So it must be aware of all existing protcols. So using a protocol extension requires to patch the lib.

There is still room for improvement.

3^rd try: Handler traits

That's what led me to the current new design (version 0.7):

• each interface defines a Handler trait, with a method for each event/request (depending on the side)
• the user create handler structs, that each implement one or more of these traits
• these handler structs are given to the event_queue / event_loop (name depending on the side)
• each wayland object is registered to a handler, whith appropriate type-checking to ensure the appropriate handler trait is implemented

The idea here is that the handler struct is the shared state, each method of a Handler trait takes a &mut self argument, allowing to access a mutable state without the need of synchronization primitives. The C libs allow me to do that, because I have the guarantee that the events are processed sequentially: there will never be two events/requests dispatched concurrently.

Also, using the appropriate trait machinery (which I won't detail here), I got a very generic setup, allowing for new protocol extensions to easily be used by the user, without needing to patch the lib. They just need to setup a build script using the wayland-scanner crate, and it'll generate a module for the protocol that can directly integrate in the existing handling setup.

This is quite symmetric, and there are very few special cases in wayland-scanner code depending on the side (client/server) the code is generated for. More code reuse, check.

The destructor safety didn't totally go away, and I had to integrate a mechanism so that all handles to a same wayland object (as I create them out of thin air for handler methods, there can be more than one) share an AtomicBool registering if the object was destroyed or not, and ignoring any attempt to send a message with a destroyed object (and returning an error signifying the destroyed state).

What could be better

With specialization, I could have made the API much more elegant. For now, there are some macros that the user need to use to make handler work, and that are there only to write impls of traits that could have been achieved via specialization, I think.

There are still some things that I see as not elegant I will try to change as soon as I find a satisfying design. First of them, the fact that to handle remote-destruction of objects, all of their methods return a result-like enum, even objects that cannot be destroyed!

What will be blocking

#![feature(static_recursion)]

I know this looks silly, but I need this feature, and I cannot do without it in the long term. The core protocol is okay for now, but the is a protocol extension that needs to be supported, it's named XDG-shell and will be in near future (or is already actually) the standard for managing the shell interaction of the client (create windows, move them, get them fullscreen, resize them, etc...).

The code generation forces me to generate a large graph of statics with pointers to each other. It's a binary representation of the interfaces and their interactions, that I must provide to the C libs to use the protocols extensions, and I can't change the format.

And the graph for XDG-shell contains a cycle.

What now?

For now, my short term goals are:

• update wayland-window and wayland-kbd to the new design
• on top of them, update glutin and winit to the new design
• then, start digging into servo on wayland, for fun, awesomeness, and intellectual interest

Also an interesting project would be to build a wlc-like crate on top of wayland-server, but this is not my priority.