Wayland: tomorrow’s graphics tech today (by way of the Raspberry Pi)
It already powers the hardware behind some set-top boxes and in-car infotainment solutions, but few realise that some development work on Wayland, the proposed display server replacement for X, is thanks in part to the Raspberry Pi…
Raspberry Pi Today recently spoke to Daniel Stone, Graphics Lead at Collabora, the open source consultancy working – among other things – on Wayland. He tells us how Wayland will be behind the graphics of the smallest embedded devices, right up to desktop PCs powered by Intel, Nvidia and AMD; how Collabora’s relationship with the Raspberry Pi Foundation has led to staggering performance improvements; and how the little British credit card-sized PC has actually played a pivotal role in Wayland’s development.
The current landscape
To help find our feet Daniel talked us through Collabora’s efforts to replace X – the current display server for UNIX-like computers and embedded devices like the Raspberry Pi – with Wayland. Graphics software is almost certainly some of the most complex and least understood in the industry, but Daniel did a great job of helping us spell out the basics.
The requirements of modern computers and devices has changed so much, that the software backbone can no longer serve its needs. X’s original design was that clients would send a description of the entire user interface to the server, which would render it for them. It worked well in the 1980s and 1990s since there wasn’t a great deal to display, but today’s multifaceted user interfaces need a more modern approach, as Daniel explains.
“Your app would start up and say ‘draw these 70 lines here, a circle here and some text over here’. But as apps have got more sophisticated the model has boiled down to ‘here’s an image I’ve already drawn – it’s my entire window, please show that’. For that and a few other reasons it’s really quite an awkward fit. You’re effectively trying to shoehorn a model where the client leads the rendering into X, which is designed to do the opposite. Wayland is designed purely for that reason – less like trying to hammer a square peg into a round hole.”
Not having to fight the model you’re working with clearly has its advantages, but other key areas Wayland promises to revolutionise our user experiences is performance and reliability. “Trying to get consistent performance out of X11… it’s guaranteed work… but it’s really not what we want,” Daniel quips.
The Raspberry Pi’s vital role
As it turns out the Raspberry Pi has been pivotal in helping Collabora validate the work Daniel and the team have been doing for Wayland. “The Raspberry Pi has been amazing in terms of keeping us honest,” he explained. “If you can get something to work smoothly on the Pi, it will essentially turn out well on basically anything else you run it on.
“Another reason it’s been good is because a lot of it has gone into using these features in the 2D display control that you can’t get out through OpenGL or similar. The Raspberry Pi’s genesis is in the media set-top box arena, where they have amazingly powerful display controllers. We wanted to unlock a load of those features with Wayland and actually make full use of them.”
With X everything is taken care of by the CPU, but with Wayland it is all offload all of that to the graphics hardware. It’s no small feat, but the advantages of this for the Raspberry Pi and similar devices are plain to see.
A freeform relationship
Daniel and his team at Collabora have actually been working very closely with the Raspberry Pi Foundation for the last couple of years. “We’ve done quite a few things for the Foundation, all directed at making the Raspberry Pi’s desktop experience more usable. When it first came out it was really quite rough and ready. If you didn’t have hands-on experience, it was really quite awkward and difficult to use.”
It turns out their working relationship is quite a freeform one: “Generally we try to identify some specific areas we can work on,” says Daniel. “Then we get really focussed on it and drain it as far as we can before we move on.”
“The first job we did with the Foundation, which caused quite a splash, was the Wayland work, where we managed to utilise this flexible display control hardware they have to deal with all the animations, re-drawing and the whole compositing of the desktop.”
Daniel’s talking about Maynard, a demo-style lightweight Wayland shell that offers faster performance and sleeker look that utilises graphical effects and animations.Test Collabora’s Maynard shell on your Raspberry Pi to see a small sample of what Wayland can do for embedded devices.
“We made Maynard minimal and lightweight. You can list the features on one hand,” recalls Daniel. “It’s got a clock, volume control, application launchers and that’s really about it. We just wanted to make something as smooth and simple as possible to use for day-to-day use, that would actually stand up.” And stand up it does, even with reasonably heavy use of animations. “We didn’t want to be obnoxious with the animation, but rather than things just appearing, they’ll slide into the screen. The fact that any of that actually happens smoothly rather than jerking horribly everywhere does a lot to validate our design decisions.”
Doing more with yesterday’s technology
You don’t need to be an industry expert to realise that the Raspberry Pi is far from cutting edge, but fewer people are aware that it was already outmoded when the first models rolled off the production line a couple of years ago. This creates many challenges in terms of performance, not least when it comes to web browsing.
“In terms of the processor, the Raspberry Pi is about as capable as an iPhone 3G,” Daniel tells us. “So you’re going back to those 2007-2008 classes of device where things like the web experience was a very different thing. You needed brutally cut-down web pages that were ugly and undemanding as possible to get anything to display.”
Since Collabora do a lot of work on Webkit too, they decided that for their next project they’d see how far the Raspberry Pi could be pushed in terms of improving the quite dire web browsing experience. They managed to successfully get Web (an unfortunately named browser from the GNOME project, previously known as Epiphany) up and running on the Pi back in December last year and the results were more than just promising.
“We put in a bunch of efforts on a few fronts,” he continues. “The first thing was immediate graphics rendering – to actually get content up onto the screen… Then scrolling performance. We also put in a lot of focussed effort behind the scenes to get rid of the unnecessary copying around of image data and excess processing that we didn’t need.”
Next, they looked at web video performance: “We used the Pi’s graphics hardware to decode the video for us and we managed to achieve 24-40 frames per second at 720p. Through HTML5 video you can genuinely watch HD movies at full frame rate. We put a long shift into that and we’re really happy about it.” And so they should be – when they started performance was rated at less than half a frame per second.Read more about the Web beta and test it yourself with three simple terminal commands at the RaspberryPi.org.
Wayland: a scalable solution
As if completely de-coupling a thirty year old mainstay in the graphics software industry and replacing it with a modern counterpart isn’t work enough, the real power of Wayland is that it’s going to be completely scalable – essentially the same graphics backbone in the set-top box, in your smartphone, in your car’s dash – right up to the dominant PC vendors that create computers many, many times more powerful than the humble Raspberry Pi.
This quite staggering feat of software engineering isn’t without it’s drawbacks, though, at least for Raspberry Pi enthusiasts that want to see Wayland on the Pi any time soon. A new Raspberry Pi user experience is still some way off and there’s still a lot of fundamental underlying work that needs to be done first. “The Raspberry Pi graphics stack is currently in a lot of flux,” explains Daniel. “It’s in the middle of having its entire driver model changed from a proprietary Raspberry Pi-specific set of graphics APIs at the low level, to the new model… the one that the desktop guys use.”
This is far from a minor change and the graphics stack is not an everyday coding conundrum, as Daniel explained. “It requires almost everything to be torn up and started fresh from line one of the source code.
The process to move the Raspberry Pi closer to that desktop model has only started in the last month or two. It has been progressing very quickly, but the last thing we want to do is push out Raspberry Pi-specific dependancies at the expense of everyone else. Once it’s all settled down we can see where it lands and go from there. At the moment we need to drive the entire project forward for everyone.”Speaking of driving forward, those able to do so are encouraged to develop with the Maynard demo Collabora put together. Full instructions on how to get started can be found on Raspberry Pi’s Github.
The future looks bright
While Raspberry Pi enthusiasts have a long wait on their hands, development of Wayland itself seems to be settling down nicely. “It’s going really well. We had a release a few weeks ago, which was a milestone in itself,” says Daniel, before telling us quite how madcap it’s been for the last few years with the huge chunks of work needed to lay the foundations.
“In stark contrast, if you look at the release notes for the version 1.5 release of Wayland and really the banner feature is that things have really settled down and there arent really any banner features! We think it’s at a really good point now for people to use, implement and ship Wayland.”
Much of this, says Daniel, has been borne out by a lot of conversations Collabora have been having with interested parties, particularly silicon manufacturers. “Implementing an X11 driver that’s stable and performant is a huge amount of work, which is why a lot of people don’t do it. These manufacturers have been looking to get out of it and in the last couple of years our discussions have centred around their interest in Wayland and wanting to know more. This year, though, those conversations have turned into ‘okay, this is definitely something we want to do, how do we go about doing it?’”