Pages Menu
Categories Menu

Posted by on Jul 1, 2014 in Features

Wayland: tomorrow’s graphics tech today (by way of the Raspberry Pi)

Wayland: tomorrow’s graphics tech today (by way of the Raspberry Pi)

It already powers the hardware behind some set-top boxes and in-car infotainment solutions, but few realise that some development work on Wayland, the proposed display server replacement for X, is thanks in part to the Raspberry Pi…

Raspberry Pi Today recently spoke to Daniel Stone, Graphics Lead at Collabora, the open source consultancy working – among other things – on Wayland. He tells us how Wayland will be behind the graphics of the smallest embedded devices, right up to desktop PCs powered by Intel, Nvidia and AMD; how Collabora’s relationship with the Raspberry Pi Foundation has led to staggering performance improvements; and how the little British credit card-sized PC has actually played a pivotal role in Wayland’s development.

The current landscape

To help find our feet Daniel talked us through Collabora’s efforts to replace X – the current display server for UNIX-like computers and embedded devices like the Raspberry Pi – with Wayland. Graphics software is almost certainly some of the most complex and least understood in the industry, but Daniel did a great job of helping us spell out the basics.

The requirements of modern computers and devices has changed so much, that the software backbone can no longer serve its needs. X’s original design was that clients would send a description of the entire user interface to the server, which would render it for them. It worked well in the 1980s and 1990s since there wasn’t a great deal to display, but today’s multifaceted user interfaces need a more modern approach, as Daniel explains.

less like trying to hammer a square peg into a round hole

“Your app would start up and say ‘draw these 70 lines here, a circle here and some text over here’. But as apps have got more sophisticated the model has boiled down to ‘here’s an image I’ve already drawn – it’s my entire window, please show that’. For that and a few other reasons it’s really quite an awkward fit. You’re effectively trying to shoehorn a model where the client leads the rendering into X, which is designed to do the opposite. Wayland is designed purely for that reason – less like trying to hammer a square peg into a round hole.”

Not having to fight the model you’re working with clearly has its advantages, but other key areas Wayland promises to revolutionise our user experiences is performance and reliability. “Trying to get consistent performance out of X11… it’s guaranteed work… but it’s really not what we want,” Daniel quips.

The Raspberry Pi’s vital role

As it turns out the Raspberry Pi has been pivotal in helping Collabora validate the work Daniel and the team have been doing for Wayland. “The Raspberry Pi has been amazing in terms of keeping us honest,” he explained. “If you can get something to work smoothly on the Pi, it will essentially turn out well on basically anything else you run it on.

“Another reason it’s been good is because a lot of it has gone into using these features in the 2D display control that you can’t get out through OpenGL or similar. The Raspberry Pi’s genesis is in the media set-top box arena, where they have amazingly powerful display controllers. We wanted to unlock a load of those features with Wayland and actually make full use of them.”

With X everything is taken care of by the CPU, but with Wayland it is all offload all of that to the graphics hardware. It’s no small feat, but the advantages of this for the Raspberry Pi and similar devices are plain to see.

A freeform relationship

Daniel and his team at Collabora have actually been working very closely with the Raspberry Pi Foundation for the last couple of years. “We’ve done quite a few things for the Foundation, all directed at making the Raspberry Pi’s desktop experience more usable. When it first came out it was really quite rough and ready. If you didn’t have hands-on experience, it was really quite awkward and difficult to use.”

It turns out their working relationship is quite a freeform one: “Generally we try to identify some specific areas we can work on,” says Daniel. “Then we get really focussed on it and drain it as far as we can before we move on.”

We didn’t want to be obnoxious with the animation

“The first job we did with the Foundation, which caused quite a splash, was the Wayland work, where we managed to utilise this flexible display control hardware they have to deal with all the animations, re-drawing and the whole compositing of the desktop.”

Daniel’s talking about Maynard, a demo-style lightweight Wayland shell that offers faster performance and sleeker look that utilises graphical effects and animations.

Test Collabora’s Maynard shell on your Raspberry Pi to see a small sample of what Wayland can do for embedded devices.

“We made Maynard minimal and lightweight. You can list the features on one hand,” recalls Daniel. “It’s got a clock, volume control, application launchers and that’s really about it. We just wanted to make something as smooth and simple as possible to use for day-to-day use, that would actually stand up.” And stand up it does, even with reasonably heavy use of animations. “We didn’t want to be obnoxious with the animation, but rather than things just appearing, they’ll slide into the screen. The fact that any of that actually happens smoothly rather than jerking horribly everywhere does a lot to validate our design decisions.”

Doing more with yesterday’s technology

You don’t need to be an industry expert to realise that the Raspberry Pi is far from cutting edge, but fewer people are aware that it was already outmoded when the first models rolled off the production line a couple of years ago. This creates many challenges in terms of performance, not least when it comes to web browsing.

“In terms of the processor, the Raspberry Pi is about as capable as an iPhone 3G,” Daniel tells us. “So you’re going back to those 2007-2008 classes of device where things like the web experience was a very different thing. You needed brutally cut-down web pages that were ugly and undemanding as possible to get anything to display.”

Since Collabora do a lot of work on Webkit too, they decided that for their next project they’d see how far the Raspberry Pi could be pushed in terms of improving the quite dire web browsing experience. They managed to successfully get Web (an unfortunately named browser from the GNOME project, previously known as Epiphany) up and running on the Pi back in December last year and the results were more than just promising.

“On the same processor now, you’re looking at doing full-fat, all javascript web pages at resolutions like 1080p.” Daniel’s clearly proud of what they’ve been able to achieve, and rightly so: the mobile Internet experience we take for granted today simply didn’t exist when the Raspberry Pi’s core components were being forged.

When we started performance was rated at less than half a frame per second

“We put in a bunch of efforts on a few fronts,” he continues. “The first thing was immediate graphics rendering – to actually get content up onto the screen… Then scrolling performance. We also put in a lot of focussed effort behind the scenes to get rid of the unnecessary copying around of image data and excess processing that we didn’t need.”

Next, they looked at web video performance: “We used the Pi’s graphics hardware to decode the video for us and we managed to achieve 24-40 frames per second at 720p. Through HTML5 video you can genuinely watch HD movies at full frame rate. We put a long shift into that and we’re really happy about it.” And so they should be – when they started performance was rated at less than half a frame per second.

Web_preview_image

Read more about the Web beta and test it yourself with three simple terminal commands at the RaspberryPi.org.

Wayland: a scalable solution

As if completely de-coupling a thirty year old mainstay in the graphics software industry and replacing it with a modern counterpart isn’t work enough, the real power of Wayland is that it’s going to be completely scalable – essentially the same graphics backbone in the set-top box, in your smartphone, in your car’s dash – right up to the dominant PC vendors that create computers many, many times more powerful than the humble Raspberry Pi.

This quite staggering feat of software engineering isn’t without it’s drawbacks, though, at least for Raspberry Pi enthusiasts that want to see Wayland on the Pi any time soon. A new Raspberry Pi user experience is still some way off and there’s still a lot of fundamental underlying work that needs to be done first. “The Raspberry Pi graphics stack is currently in a lot of flux,” explains Daniel. “It’s in the middle of having its entire driver model changed from a proprietary Raspberry Pi-specific set of graphics APIs at the low level, to the new model… the one that the desktop guys use.”

The last thing we want to do is push Raspberry Pi-specific dependancies

This is far from a minor change and the graphics stack is not an everyday coding conundrum, as Daniel explained. “It requires almost everything to be torn up and started fresh from line one of the source code.

The process to move the Raspberry Pi closer to that desktop model has only started in the last month or two. It has been progressing very quickly, but the last thing we want to do is push out Raspberry Pi-specific dependancies at the expense of everyone else. Once it’s all settled down we can see where it lands and go from there. At the moment we need to drive the entire project forward for everyone.”

Maynard desktop development github

Speaking of driving forward, those able to do so are encouraged to develop with the Maynard demo Collabora put together. Full instructions on how to get started can be found on Raspberry Pi’s Github.

The future looks bright

While Raspberry Pi enthusiasts have a long wait on their hands, development of Wayland itself seems to be settling down nicely. “It’s going really well. We had a release a few weeks ago, which was a milestone in itself,” says Daniel, before telling us quite how madcap it’s been for the last few years with the huge chunks of work needed to lay the foundations.

“In stark contrast, if you look at the release notes for the version 1.5 release of Wayland and really the banner feature is that things have really settled down and there arent really any banner features! We think it’s at a really good point now for people to use, implement and ship Wayland.”

Much of this, says Daniel, has been borne out by a lot of conversations Collabora have been having with interested parties, particularly silicon manufacturers. “Implementing an X11 driver that’s stable and performant is a huge amount of work, which is why a lot of people don’t do it. These manufacturers have been looking to get out of it and in the last couple of years our discussions have centred around their interest in Wayland and wanting to know more. This year, though, those conversations have turned into ‘okay, this is definitely something we want to do, how do we go about doing it?’”

  • Pingback: Wayland still a way off for the #RaspberryPi, but getting there! | Raspberry Pi Pod()

  • Raphael Sanches

    That so called future should actually be now!…

    … Really Funny to see someone from the Wayland Project doing some PR to compensate the fact that MIR has progressed in 2 months what took Wayland 2 years! … after all the MIR buzz people remembered that once upon a time there was a project called Wayland, and even though that project showed no progress there was funds being injected into it !!! …
    … Really Disturbing to see that the focus has been so far on “piece of cake” Pi drivers instead of the big/huge/endless problems that NVIDIA / AMD drivers are!
    … Really Shameful to see a “serious project” using the “… Pi is so easy even a 7 years old can use…” as their inspiration!

    P.S. Can’t wait until 2030 to see how a real Desktop PC powered by Wayland can handle incredibly complex games… like… Tetris ! … ; )

    • daniels

      Mir builds on the years of effort we spent building DRM/KMS, EGL/GLES support in Mesa, libhybris, GBM for buffer management, libxkbcommon, XWayland, and a hell of a lot more. In a few cases, they actually just copy and pasted our code and changed the names.

      Which is fine: they’re welcome to do that. But they wouldn’t have been able to do it without all the years of groundwork we did to make it possible to use _anything_ other than X11.

      Also, we’re shipping Wayland in phones, set-top boxes / media devices, and Fedora is shipping GNOME with Wayland. Mir has not shipped anywhere yet.

      • Raphael Sanches

        Well Daniels… exactly!

        … “they actually just copy and pasted our code” – well… maybe you guys could do the same and catch up!

        … “we’re shipping Wayland in phones, set-top boxes / media devices” – could you be kind and share more information with us about those devices?

        …”Mir has not shipped anywhere yet” – Canonical will ship more than just MIR by the end of the year, actually a whole OS, which means they have been busy working on more than just one project …. and still were able to leave Wayland behind!

        …”Fedora is shipping GNOME with Wayland” – I don’t think that those generic Distros (DE + Wallpaper Pack + Cool Logo) used by 3 people who really enjoy pain and suffering actually matters that much, since when somebody needs stability you know which Distro to go to…. ; )

        P.S. Really surprised to get a reply from somebody from Wayland Project (supposed) on the same day!… I was expecting that only in 2030, right after Wayland finally reaches a stable release (Open Source Drivers only) ! ! ! ………. : )

        • AmericanHoser

          From the words you say, it seems you must work pretty hard at being ignorant. Either that, or you’re paid to act that way. I’ll be the one to say it, though, since the Wayland devs seem to want to stay polite.

          > … “they actually just copy and pasted our code” – well… maybe you guys could do the same and catch up!

          The problem with that idea is that Mir doesn’t have much in the way of original code to copy. All of Mir’s big “wins” were taken from other projects. Like running android drivers with libhybris, which was a proof of concept made for wayland nearly half a year before Mir devs swiped it. Mir’s just a hodgepodge of existing projects with a bit of glue.

          The reason for the genesis of Mir was an economical one, not a technical one. Despite misleading statements to the contrary. All of the Mir complaints around Wayland were proven incorrect. In reality, it just gives Ubuntu licensing control over low-level components. Which makes it easier to sell licenses to phone distributors.

          > …”Mir has not shipped anywhere yet” – Canonical will ship more than just MIR by the end of the year, actually a whole OS, which means they have been busy working on more than just one project …. and still were able to leave Wayland behind!

          Nope. Fedora allowed users to start trying Wayland in 2013. Wayland has been accessible for a while. Wayland is in a more complete state by far, and it actually has community support. Most of the old X devs are now working on Wayland. Unlike Mir, which has managed to alienate most of the community for misleading statements which people like you eat up.

          Wayland is currently in a more usable state, and both should be user defaults in one or more distros in 2016. Not this year. Shuttleworth stated that the Mir release would be delayed until 2016 back in March. They might do a preview before then, letting users try it out. But that still puts them behind Wayland, who started making Fedora previews a year ago.

          In fact, in 3 months when Fedora 21 is released, most of what a user needs will be available for use in Wayland if those interested want to try it out. Which I found out with about 10 minutes of Googling. You should try Google in addition to the Ubuntu forums. There’s a lot more information out there.

          > since when somebody needs stability you know which Distro to go to

          No one in their right mind would ever pick Ubuntu for a stable distro. They would pick Debian, which does most of the work Ubuntu rides on, or CentOS, which is closely tied to Fedora. Fedora is actually quite popular, though I don’t use it myself. Ubuntu did a good thing in making a Linux install more user-friendly, but that is it’s sole claim to fame. Not stability. Nor development innovation.

          > I don’t think that those generic Distros (DE + Wallpaper Pack + Cool Logo) used by 3 people

          You realize that far more work goes into Fedora than Ubuntu, right? Ubuntu actually uses Debian packages, and has made various roadmap decisions (such as switching the init system) based solely on what Debian is doing to reduce the amount of work that they do. If anything, Ubuntu is a better fit for your statement, since it’s essentially Debian with the Unity DE. Is accusing people of your own failings a popular method of insulting people?

          Honestly, if uneducated and ill-mannered people like you are the users of Ubuntu, I’m glad I’m not a part of it anymore. I left Ubuntu over 6 years ago and you’re only convincing me that it was the right decision.

          > … “we’re shipping Wayland in phones, set-top boxes / media devices” – could you be kind and share more information with us about those devices?

          Sailfish OS (a smartphone OS by Jolla) uses Wayland.

          Tizen (in smartphones, car IVI systems) uses Wayland. Wayland is actually hugely popular with automotive interfaces. Check out the radio/video bling in a car dash or the back of a modern SUV – it’s likely to use Wayland.

          Gnome, KDE, and Enlightenment DEs all support Wayland. Support is not complete, but it’s in a good state. None of them have added Mir support.

          Most of the major windowing and game toolkits either support Wayland or are looking into it. Examples: Clutter, EFL, GTK, Qt, and SDL. The only way Mir will get most of that is if the Ubuntu devs add it themselves.

          So, now that we’ve had this little talk, can you please stop being rude to the people who are driving the innovation? I mean, you can like Ubuntu all you want, but I’m sure even Ubuntu devs would appreciate it if you stopped hassling the people they’re “borrowing” code from. And I, for one, am grateful to the people who are doing the actual work.

          • Raphael Sanches

            Dear Unicorn Lover…
            I am sorry but I simply don’t buy all this cheap PR in this article stating that everything is beautiful and perfect when is not!

            - Innovation… Wayland is late!… Linux is about to enter the Gaming Industry and Wayland devs are proud that they finally found a way to minimize windows!…. Come on!… Really?… I wonder that if in 2016 we will have Open Drivers compatibility, It means that in 2030 we will have Proprietary Drivers compatibility!
            - Sailfish/Tizen!… that is all Jolla/Samsung devs work on top of Wayland (deadlines/financial expenses/no money going down the drain over stalling)!
            - Ubuntu… I wish to use other distributions but Fedora/Suse and all that crap just pushes me back to Ubuntu whenever I try… Have you ever try to install Nvidia/ATI drivers on Fedora/Suse?… or Chromium?…or Flash?…or anything?… Crap Crap and more Crap…. but don’t worry about Fedora, it only exists so Red Hat/Suse can prove that they are a Open Source Company (keep up appearances)…
            I am a person who believes that computers should do work and fix things for me and not the opposite!

            … My intention was to sarcastic and ironic on my comments, not rude, since everyone now is kissing Wayland’s ass I thought somebody should look at the not so perfect aspect of all this… Not is perfect or even close to finished and ready, Wayland will bring us as much problems as solutions….

            … Now AmericanHoser, you can go back to your rainbow, you probably have a lot to fix in your Fedora installation… ; )

          • AmericanHoser

            Your ignorance, astounding as it is, is dwarfed only by your inexplicable belligerence. I’m guessing that reading is something you do only under duress.

            > I wonder that if in 2016 we will have Open Drivers compatibility, It means that in 2030 we will have Proprietary Drivers compatibility!

            Open driver compatibility is already mostly there. It should be complete this year.

            Plans are being made for proprietary driver compatibility. LibHybris was originally a Wayland proof of concept. There are discussions involving the best approach to dealing with it. Furthermore, reverse engineered drivers such as etna_viv and lima are already doing quite well, and should be completely usable before 2016. Etna_viv already supports OpenGL, afterall. Which you would know if you ever bothered to research anything before opening your mouth.

            > Wayland is late!

            Wayland may be late in your eyes, but it’s still earlier than Mir despite having accomplished more, and will continue to be so. Which you would also know if you bothered to look up the timelines.

            > Ubuntu… I wish to use other distributions but Fedora/Suse and all that crap just pushes me back to Ubuntu whenever I try. Have you ever try to install Nvidia/ATI drivers?

            Yes, I’ve worked with more than 10 different distributions on four different platforms (x86, PowerPC, ARM, MIPS), as well as compiled from source using OpenWRT and Gentoo. It’s not that hard for most setups. Fedora wasn’t that bad, and had a rather usable default install. And Arch, for example, has Wiki pages and custom repositories for the proper X versions and dependencies. Your inability to migrate does nothing but showcase your own shortcomings.

            > Not is perfect or even close to finished and ready

            That’s why it’s planned for the default in 2016. You can run it now, but 2016 is when it will be ready out of the box. Shuttleworth has given a similar timeline for Mir, though I don’t think they’ll make it. He’s delayed it three times already.

            > Wayland will bring us as much problems as solutions.

            As does any new technology. Most of them have been ironed out.

            > … Now AmericanHoser, you can go back to your rainbow
            > Dear Unicorn Lover

            I’m so totally hurt that you’re calling me names now. Really. Crying into my big rainbow pillow.

            > you probably have a lot to fix in your Fedora installation… ; )

            I’m running Arch, these days. It’s apparently where us elitist assholes who are competent enough to google before making fools of ourselves live these days. And it’s been running fine for years. Unlike Ubuntu, where you have to reinstall to upgrade every 6 months.

            Seriously, though, the point is that most of the distributions are pretty good. Ubuntu is not the best. It’s just where people start out until they learn that there’s better out there. I’d suggest Debian to you, as Ubuntu is mostly Debian work anyway.

            Mir might succeed. If so, more power to them. But it’s not looking good at the moment. And Wayland is. The Wayland devs are the people who maintain X. They know Linux graphics, and they know its shortcomings. They’ll make it happen.

            > My intention was to [sic] sarcastic and ironic on my comments, not rude
            > Really surprised to get a reply from somebody from Wayland Project (supposed) on the same day!… I was expecting that only in 2030, right after Wayland finally reaches a stable release

            No, you hit rude pretty squarely. All of your comments were mocking Wayland pretty mercilessly. Just because you thought you were being funny when you wrote it does not make it less rude. Not to mention you were completely off-base about the topic.

            Which is the main reason I commented in the first place. Your misinformed and caustic comments were likely to mislead people entirely about the state of affairs from the supposedly safe harbor of sarcasm, the lowest form of wit. Good luck internetting in the future. I have no more time to spend on educating you.

          • Raphael Sanches

            Dear AmericanHoser

            Your fancy words and long comments don’t change the fact that:

            - Until Wayland is able to run high demanding 3D Games with NVIDIA/ATI Drivers it is just a scratch on a napkin, just a worst version o X with a longer name! … that is called REALITY!…

            - To anybody who’s jumping into Linux: YOU ARE ABOUT TO HAVE ENDLESS BUGS, PROBLEMS, ISSUES, NIGHTMARES, WAISTED WEEKENDS WITH YOUR VIDEO CARD … that is called REALITY!…

            - The Wayland developers should be sucking NVIDIA developer’s dicks right now!… Wayland should be built according to NVIDIA!… that is called MARKET REALITY!

            Even though we Linux users, who represent less than 1,5 %, would like to believe that if Linux is good enough for us, is good enough for anybody!… if I can use the Terminal, anybody can! … the Companys who’s opinions actually matters to the technology industry don’t see things this way!…

            Samsung / Jolla is only using Wayland because is free, as in “free beer”, which means MORE PROFIT!

            Probably the guys at the OSu read all this illusions because just last week they announce a kickstarter to built a brand new and modern OS featuring…. Wayland!… because X is too old!… LOL !!! Later they came back to reality and changed it:
            http://www.operatingsystemu.com/

            Since you are an “educator”, I would really apreciate if you “educate” me about the aspects of Wayland I know little about and that YOU AVOIDED in your long and delusive past comments:

            - Who’s financing Wayland?

            - How much has been spent on the Wayland Project?

            - How much more will be necessary to complete the project?

            - Did you take your medicine this morning? … ; )

            - I thought you liked Unicorns and Rainbows !?!

            P.S. You are the type of Linux user who believe that as long as you are able to reach terminal everything is fine! Its good enough! …. I am the type of Linux user who believes Linux deserves, can and should offer more than that!

          • Dan Wolf

            I’m a pretty big fan of my Ubuntu install. I like Crunchbang (a simple Openbox on Debian), but the reason I like Ubuntu is the same as the reason I like Raspberry Pi. It’s just very prolific and comfortable with tons of documentation put together in formats that are easy to understand for people who know technology but are not Linux admins (professionally I use Win and IIS, sadly). I also happen to enjoy Unity. However, I do think it’s a little stupid how Mark went about the announcement to split off Mir. He’s welcome to do it, and I don’t think splintering it damages anything. I’m a programmer, and larger dev teams often do NOT speed up a project in the least. There is the trouble with getting proprietary drivers for both now, but that will probably iron itself out eventually. Really the problem is how much drama is thrown around over the issue. Both teams do good work and if something is copied over or not it isn’t bad. It’s free software. This type of inflammatory posting is not professional or mature. I’m pumped about the next Unity and equally pumped about using Wayland in Maynard (GASP)!

          • AmericanHoser

            Sorry, I can’t educate the willfully ignorant. Your ego is too big for you to learn anything. Best of luck.

        • directhex

          > could you be kind and share more information with us about those devices?

          Recent LG TVs. Over a million units shipped.

          • Raphael Sanches

            First Tizen phone postponed indefinitely…
            http://linuxgizmos.com/first-tizen-phone-postponed-indefinitely/

            … LMFAO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

          • daniels

            Tizen Phone doesn’t use Wayland.

          • Raphael Sanches

            OMG!

            Wikipedia:
            Tizen up to 2.x supports Wayland in in-vehicle infotainment (IVI) setups and from 3.0 onward defaults to Wayland.

            … LMFAO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! … Even harder!

          • daniels

            These phones were not 3.x. Tizen IVI already ships, as does GENIVI, everywhere. Anyway, enough trolling.

          • Raphael Sanches

            Which Phones are you talking about??? Tizen Phones don’t exist!

  • Pingback: Wayland: tomorrow's graphics tech today (by way of the Raspberry Pi) | Techbait Tech News()

  • Pingback: Links 4/7/2014: E19 Alpha 2, KDevelop 4.7.0 Beta | Techrights()

  • Raphael Sanches

    OMG!

    Wikipedia:
    Tizen up to 2.x supports Wayland in in-vehicle infotainment (IVI) setups and from 3.0 onward defaults to Wayland.

    … LMFAO !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Even harder, daniels !

  • Pingback: Wayland: tomorrow's graphics tech today (by way...()

  • Pingback: Gregory Smith()

  • Pingback: мини кредит()

  • Pingback: Wayland: tomorrow's graphics tech today (by way...()