In the Linaro Multimedia group we hosted 4 sessions about audio on the various member company boards. These included, STE, TI, Freescale, and Samsung. It was a great opportunity to meet with the landing teams who work to keep the kernel updated, push patches upstream in the kernel and in some cases enable the multimedia stack. I say some cases as not all drivers are open from board to board. We were able to talk about common problems, how we can help each other and most importantly keep multimedia working on Linux and Android.

Some themes emerged from the meetings for which we are able to take acti. First the teams all need some sort of easy to use command line tests to validate things are functional. Second the sound configuration system needs to change to be less brittle.

Over the course of the week the STE team is to be commended as audio was made to be working on snowball. They were running xbmc by the end of the week which is great accomplishment.

On Tuesday morning I gave one of the 15 minute plenary talks. Hope the talk makes it up to youtube where the other talks are starting to appear. In short the talk was about audio and accelerated video aren’t nice to have anymore, they are requirements and if they aren’t working, we need to consider the build broken. www.youtube.com/user/LinaroOrg/feed

Beyond Audio sessions there was a meeting about codecs and NEON optimizations. VP8 is high on the wish list as many would like to have good demo capabilities using Google+ hangouts which of course utilizes VP8. WebP the newer still image format also appeared on the list. Libav is still a popular item however since it is not on Android this was an item for discussion as far as how to maximize benefits to Android besides just Linux.

There was also a gstreamer session. The discussion largely centered on the release of gstreamer 1.0 and the efforts and issues that will need to be considered as part of the upgrade.

dma_buf was also a discussion point in it’s own set of sessions. From what’s next to Android discussions and related. I didn’t make it to most of those sessions unfortunately. With Jesse Barker and Rob Clark driving those efforts it continues to move forward. With the acceptance into the 3.3 kernel, it’s exciting times. Further Rob had a demo at the Friday demo session of a camera using dma_buf. Exciting stuff.

During the hacking sessions, I upgraded libjpeg-turbo to 1.2. BenjiG was able to get even more test cases running in LAVA. Feng was working furiously to make the 12.02 release with stability updates for alsa and pulse audio for Panda and imx53. Kurt while not in attendance was working on UCM for Android.

Another high point of LC was the use of Google+ hangouts over the air. It worked and allowed remote participants to be part of the sessions.

Why go to a Linaro Connect? Linaro Connect is part planning the future of ARM and part hacking sessions. For an engineer it’s a high bandwidth, high productivity environment. You can watch world cup soccer on TV, you can watch the super bowl on TV, but being there is an entirely and more rewarding experience not only for the individual but for the ARM community. Sure one can attend a session or Google+ hangouts but you can’t sit next to someone at a hacking session, you can’t grab someone in the hallway, you miss the shared experience and likewise the community misses the shared experience of having you there. Ideas, designs, conversations go in far more interesting directions with you in the audience.

Hope to see you in Hong Kong!

Even with the short cycle work was accomplished and I’d like to share some of the highlights with you.

Audio

One of our major goals for this cycle was the return of working audio. This is a fairly difficult area especially on Linux as the between the kernel, alsa and pulse audio, there’s plenty of room for regressions to occur. In past months we’ve been bit strongly by those regressions such that when it comes to audio on the Linaro LEB, I’ve taken a very inbox zero approach. It’s all broke unless proven otherwise. Let’s fix it!

So as a team we focused on audio on Panda, Panda ES and imx53. All of these boards have output via an audio jack as well as out HDMI when HDMI is used for video out. Wei Feng along with the respective ti and freescale landing teams worked hard to address the kernel drivers, ucm configs which are packaged with alsa and updates to pulse audio to support ucm.

With Panda there is an existing bug where DVI video is busted. Because of this if you are validating the audio jack you have to set amixer settings appropriately so when using speaker-test or aplay you’ll get output.

Here we are on release day and audio is working for our aplay and speaker-test tests. It’s a start but not quite a whole perfect user experience. We do see bugs with GUI tools that sit above. Are the bugs in the tools or in lower portions of the audio stack, that’s a good question. We know we have a start. A line in the sand which we will not back from but clearly more work ahead.

Kurt is working on the audio stack for panda on Android. On Jan 6th the Android kernel team delivered the needed support for audio. It’s a start, we look forward to enabling ucm as well as other audio stack elements in the days and weeks ahead.

Codecs

Mans started implementing optimizations for libav involving vp8.He’s a master coder and we look forward to his upstream submissions sometime in February but probably after LC since Linaro Connect and ELC are the first part of February.

Rony worked on the NEON optimized version of speex. It is part of the Linaro Android release now. Good job Rony! Rony has also worked to become on of our Lava experts.

I worked on libjpeg-turbo for android. I pull in some upstream updates fixing up the android specific patches. I then pulled in some existing qualcomm optimizations of 565 and 8888 encode and decode. I had to rework the patches a bit to get them to fit into libjpeg-turbo but in the end it appeared to be well worth it yielding about a 100% performance improvement in skia-bench.  There are some STE patches that accomplish the same that were recently given to me. I am going update the STE implementation so it fits with libjpeg-turbo and hold a bake off. Winner goes upstream.

UMM/Video

Rob added eglImage rendering support to qtmobility. This dramatically improved HW video decode so for the linaro-tv xmbc image and Ubuntu TV efforts on panda, it looks spectacular. Ricardo Salveti wrote about this in his blog, including demos. Be sure to check it out at : http://rsalveti.wordpress.com/2012/01/06/hw-video-decode-and-xbmc-ubuntu-linaro/

Kan worked on xbmc. Specifically he’s looking at a way to improve the way video is played on xbmc using gstreamer as well as general improvements for xbmc that make it really shine on arm hardware. We’ll have a demo at Linaro Connect in February.

Benjamin worked the the continuous memory allocator debug blueprint. This is part of the larger UMM (unified memory manager) effort that Linaro is leading. A special shout out to Benjamin for becoming and helping out others with getting testcases into Lava. Lava is our automated test system. It new to Linaro and having someone take the time to education and help others is really valuable.

Finally

I hope you’ll agree it’s a good set of accomplishments for a significantly shortened cycle. I can’t thank the team enough for their efforts. Look for more accomplishments in 12.02.

The wiki page that holds all this information but in a much more raw form can be found here: http://wiki.linaro.org/TomGall/DraftPlan/2012.01Plan

Our 12.02 plan is starting to take form here : http://wiki.linaro.org/TomGall/DraftPlan/2012.02Plan

Android (including the latest ice cream sandwich release) uses the old and quite crusty libjpeg library. This library while functional is missing a great deal of optimization. The libjpeg-turbo project (http://libjpeg-turbo.org) is a souped up and more importantly drop in replacement API compatible version of libjpeg. It is compatible with both versions 6.x and 8 that are in wide use across many a distro. However because libjpeg doesn’t have simd (Single instruction, multiple data) optimizations using NEON on ARM for instance, distributions have been pitching libjpeg for libjpeg-turbo.

At Linaro our Android Ice Cream Sandwich for instance replaces libjpeg with libjpeg-turbo. Likewise we recently worked with Ubuntu and as a result Precise (version 12.04) now includes libjpeg-turbo.

So what kind of performance bump can one see by switching from libjpeg to libjpeg-turbo?

https://wiki.linaro.org/TomGall/LibJpeg8  includes lots of raw numbers on intel and ARM machines. In short performance improvements measured by tjbench  are on the order of 200% to 300%.

libjpeg-turbo doesn’t include support for Android. So earlier this year, we ported libjpeg-turbo so it would include Android dependency on the aforementioned  nonstandard 8888 and 565 formats. This was a good first step however, no work had been done to optimize for 565 or 8888. Thus the performance for 565 and 8888 was about the same for libjpeg and libjpeg-turbo.

I’d like to see Android switch to use libjpeg-turbo so this week it was time to do some optimization that would give credence to that desire. Optimizations that can be measure with skia_bench would be the way to go.

Hack. Hack. Hack.

The results? A comparison of android’s libjpeg,  libjpeg-turbo without 565 or 8888 optimizations and libjpeg-turbo with 565 and 8888 optimizations can be found at : https://wiki.linaro.org/TomGall/SkiaBenchNumbers . Smaller numbers are better.

Half the time not bad or put another way, the newly improved libjpeg-turbo is 2 times faster than the old libjpeg! I’m sure you’d like that improvement on your Android phone or tablet!  I would!

Now we get to another part of the story that is important also reflects another aspect of what makes Linaro an important leader in ARM for Linux and Android.  You see the optimization code for 565 and 8888 already sort of existed. It was sitting out in a git archive more or less gathering dust. The essential step of putting it together with the already optimized libjpeg-turbo hadn’t been done. The code also hadn’t been pushing upstream to the communities that would most benefit. In Linaro we want Linux and Android on ARM to shine.

While it’s all put together and it works, that’s great. Now comes the most important steps, getting google’s Android engineers (and perhaps cyanogen mod too!) to accept it so that all might benefit. That’s the bar for success we aim for at Linaro and we will succeed.

Code : git://git.linaro.org/people/tomgall/libjpeg-turbo/libjpeg-turbo.git

Pull from the android branch.   git checkout -b android origin/android

Have fun!

• Voice will drive new user interface innovation in 2012. Android will do something to try and catch up with Siri but  include ads in some fashion. Apple will open up the Siri API so that 3rd party developers can take advantage of it.
• Mobile app authors will continue to make more money on iOS than Android.
• Windows 8 on anything but intel will flop. PC sales will continue to slow as tablets will take the place of PCs for some.
• RIM is doomed. I expect tar, feathers and pitch. It won’t end well.
• Nokia will continue to free fall.
• We will see significantly increased display resolutions in the tablet space. The iPad will gain a retina display and Siri. There will be some sort of android based tablet that will follow up with that kind of resolution as well. Likewise in the workstation/laptop space with thunderbolt going onto all intel motherboards come mid year, thunderbolt will take off and display resolutions will fly by 1080p.
• Android based tablets will continue to fail.
• Amazon will evolve the fire and follow on products to the fire. Like Apple they will pitch their offering as an integrated solution with apps, books, movies and music. It’s not about the processor anymore. Their sales will be good.
• Apple will release an updated Apple TV but not a TV. It’ll include the ability to run apps, paired with the networks (channels are apps, apps are channels) and it will include Siri voice control. GoogleTV will continue to go no where.
• This will not be the year of the Linux Desktop.
• WebOS development will gain a following in open source circles once the source is released. It will be ported to arm development boards. The device manufacturers will gain interest. However none will ship a phone based on it this year.
• Use of the GPL/LGPL will continue to wane. Use of BSD/MIT/Apache licenses will continue to rise.
• Server Linux on ARM will for the time being be experimental. However it sill start to gain a following as ARM server hardware will continue to evolve.

• Will ChromeOS from Google be an interesting player, will it merge with Android, and will it replace Windows on hundreds of millions of desktops? No / Yes / No

• Will Android devices surpass those from Apple? Probably, but it’s a bad question. The better question is will Android surpass Apple’s quality and usability to make it a better product. No.

• Will one tablet maker emerge that will clobber the iPad in market share? No

• Will some flavor of Windows be more significant than Android on tablets? No, nor will windows on a tablet surpass iOS

• Will we see more open source apps on the most popular smartphone platforms? No. This is another poorly worded question. Who wants to use emacs on a smartphone? Developers want to make money. That’s the whole point and that’s why iOS and to a limited extent Android apps are successful. What’s the motivation, whats the ITCH for open source apps on smartphones? There isn’t any save for the hobby of bringing the familiar into the platform. Linux is about making a great operating system, not about making great apps.

• Will Linux gain further market share as people continue to flee from Solaris and install new servers for new applications? Sure, but in the end it’s just a trade of Unix for Unix. Yawn.

• Will there be more lawsuits around the use of open source in smartphones? Yes, but it’s not the open source developers that are driving the conversation here nor do they hold the poker chips. It’s the big name companies, Linux is just a pawn and that’s problematic.

• Will Windows Phone 7 beat out Android phones or iPhones? Doubtful

• Will LibreOffice pass OpenOffice in downloads? The better question is, Will LibreOffice be a major source of innovation in the word processor, desktop office suite space in 2011? No.

• Will open source virtualization via KVM start to gain market share against VMWare and Microsoft HyperV? Yes.

• Will the “open cloud” become more significant and more widely implemented? No. The cloud has been defined by the amazons, googles and such. It’s not a question of open. It’s a question of what those leaders in the market will do. There is no open cloud.

• Will Windows Internet Explorer continue to lose market share to Firefox, Chrome, and Safari? Yes.

• Will Diaspora replace FaceBook? No.

• Will any open source system replace WordPress, Drupal, and Joomla in the top three of open source content management systems? No.

• Will 2011 be the Year of the Linux Desktop? The better question is Will Linux gain market share in the Desktop market? The answer continues to be No. I disagree completely with Bob Sutor that the question is indeed VERY relevant. Because Linux continues to gain no ground here, if not indeed lose ground it’s a very significant ding to Linux. Desktop Linux was driving what little usability there is in the open source ecosystem. I predict Desktop Linux use will fall in 2011 and that Android will drive open source usability in so much that the Meegos, kwin, gnome-mobiles will follow but not drive open source innovation. I believe there is a slight chance that a new mobile operating system which will be a blend of Linaro, Meego and other bits could happen and find success.

• Will Linaro become a major factor in arm Linux in 2011. Yes. Further I believe that Android will start to utilize parts of Linaro and I believe the Meego project that runs on arm hardware will evolve if not fracture completely away from Meego to use Linaro. (If not Meego will be a complete flop and die by EOY since Linaro cares about ARM, Intel does not)

As mentioned Meego died. Android picking up pieces of Linaro… stay tuned .

Of note for Oneiric Linaro images, we’ll see a new community image added to the mix named linaro-server. To start it is a LAMP stack for arm. The image weighs in at about 120 Meg compressed. I fully expect we’ll flesh this out with more useful server goodness in time as makes sense for running on an arm server. I don’t know about you but I can’t wait to get my hands on an A15. Checkbook is standing by.

We’ll see a new insanely smaller image that is nothing more than a kernel and a busybox initrd. As to it’s name? Well I think we probably need to build some consensus, but pico was thrown out as one possibility.

The old friends of alip, ubuntu-desktop, and developer will continue on. In addition the graphical-engineering and multimedia-engineering images will continue tho I expect some adjustments so that those images pick up works in progress much quicker from the work groups in question.

Alip will have the ability for all of it’s packages to be cross built. This might cause a refactoring of the contents of the ALIP image. I would expect the images to get even smaller and perhaps even switch away from xfce to something fairly minimalistic but still good enough to run a web browser.

Stay tuned!

But the part that made me give pause because this problem has been around for decades was that when it comes to developing code for Android or iOS, well, it would seem by these numbers not many developers are using open source software to create their end user software.

The article states that 71% of Android and iPhone apps containing open source failed to comply with the four obligations of the open source licenses that OpenLogic analyzed.
Out of the 635 apps scanned, OpenLogic identified 52 applications that use the Apache license and 16 that use the GPL/LGPL license.

So at this point in the show one should point out that there are 350,000 apps in the iOS App Store and 250,000 apps in the Android store. We don’t know how they chose the roughly .1% of apps to scan. But isn’t the better question to all of this, why isn’t there more open source software in use in mobile applications? Only 10% what was scanned actually uses open source software in some capacity. Ouch.

Apple doesn’t exclude it. WordPress uses GPL stuff after all. VLC did until some of the copyright holders of the GPLed code they did use in their app contacted Apple and forced the app to be removed because they didn’t want it in the iTunes App Store. (Guess the copyleft part of the license where the next person has the same rights as the person who receives the code, you know that founding principle of open source doesn’t universally apply )

Android certainly doesn’t exclude open source software either.

Here’s a better reason. Remember when DOS, Windows, Amiga or the Mac got started. You know, all those microcomputers … I use that term explicitly because it reminds you that they were microcomputers as compared to the mainframe or small mini compters they were replacing. Did a whole bunch of mainframe software get ported to those boxes? Some but there was a whole explosion of new software to serve the new paradigm, even more so in Windows, Mac and the Amiga because the 80 column universe had evolved to something mouse based. Why use vi to edit your term paper when a GUI word processor gave you WYSIWYG?

We’re seeing that again in the mobile space. It’s a new usage paradigm. Yes sure, Android has a Linux kernel and yes it does have system libraries and things taken from various open source BSD projects. Same for iOS, BSD Mach kernel, lots of BSD library plumbing. But getting beyond that plumbing, to the interesting stuff, the apps that you interact with that’s where the action is. And in that way that’s where the open source universe has a problem. We, the open source universe, have a source base that’s engineered for a past era. We didn’t invent the Post PC era end user story and we’re not driving the post PC era end user story so our relevance is under pressure.

It gets worse….

The vast majority of users never cared what the license was of their software. It was a very rare Mac v Windows v Linux discussion that would be won by that talking point.

The developer landscape is different in that a higher representative sample care about open source and the concept of an open source license. But in this mobile space it would appear that the numbers that “care” has been reset.

This shouldn’t be surprising if as I assert that open source software isn’t engineered for mobile users. Or put another way, if open source software was solving mobile developers problems, wouldn’t the scanning that was done have revealing more open source software in use?

And now we’re to the heart of the matter, if you believe as I do that mobile is that next era in computing, then we as open source developers have a great deal of coding to be do. We’re several years behind iOS and Android. We need to decide where and how we want to be relevant. Are we add ons to existing mobile operating systems, WP7, iOS, Android or are we going to innovate in this space and sit side by side with the others and create our own ecosystem?

Remember the existing OSes don’t need new widget sets. They don’t need new rendering engines. They certainly don’t need vi. They’ve already created a mobile GIMP and countless other apps that people love and that the authors of made a bunch of money and did it all without any open source code.

Maybe the open source universe just wants to work on kernels and system libraries and that’s it. Leave everything else to proprietary solutions. That’s basically Android. I hope that’s not the case.

Simple to assemble, drop SATA drives into drobo.

Install of their dashboard on Mac OSX requires a restart. Any app that requires that OS to restart just to start up some new services probably should rethink their design. Call me old school unix but I don’t like to restart that often.

Making shares again, very easy. If you’re going to use any of the shares with time machine you have one little extra box to click, very simple stuff.

My only ding against the whole package is the data transfer rates. Over the course of approx 24 hours only ~400 gig of data had transfered. For a device hooked up to my gigE backbone and the only other machine on said gigE backbone is the box shipping data to the drobo I would have expected a much better result. In comparison 30Meg/s for USB2 would transfer ~108Gig/hour. Shipping 30Meg a sec over a quiet GigE is more than reasonable.

So all in all if you want a drop dead simple data backup system that raids hard drives drobo certainly fits the bill. If you want it to be fast… well best look elsewhere. 6 cheap USB disks raided and redundant via OSX would be both faster and cheaper.