Patchwork Chromium acceleration

login
register
mail settings
Submitter Eric Nelson
Date April 1, 2014, 7:22 p.m.
Message ID <533B11E6.5040103@boundarydevices.com>
Download mbox | patch
Permalink /patch/69961/
State Not Applicable
Delegated to: Otavio Salvador
Headers show

Comments

Eric Nelson - April 1, 2014, 7:22 p.m.
Thanks again Carlos,

On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
 >
 > <snip>
>
> Back then we needed some hw decoding quickly, so we did not look further
> into this, since we had spent a lot of time getting the 2D acceleration
> stable already. I could only briefly look at the exynos accelerator
> code, but it does look like the right direction, although the VPU uses
> physical addresses directly instead of dmabuf FDs. I am currently
> writing a frontend for the libfslvpuwrap, which takes care of certain
> complexities (like being able to pass userdata pointers to frames, to
> make it possible to associate input and output frames, which is
> essential for correct timestamping, especially when things like h264
> frame reordering are active). This frontend will hide these things in a
> future gstreamer-imx release to make the decoder code clearer and easier
> to maintain, and is intended to be reusable, for example for Chromium
> integration, or XBMC. Beyond that, my patches are probably not that
> helpful anymore, since they were based on the vpx decoder way of
> decoding, and I agree that the exynos code is a better starting point.
> Nevertheless, I attached the patch. Keep in mind that it is very basic,
> old, and wasn't further developed on because it was "good enough" for
> the project.
>
> But here are some additional notes from our efforts to accelerate
> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>
> 1. We started the google-chrome binary with these switches:
> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
> --enable-accelerated-2d-canvas
>
> 2. To make building Chromium easier, we switched to component build (by
> default, it is all linked into one enormous binary). I attached a patch
> that was necessary to fix some minor issues. Perhaps it is not necessary
> anymore.
>
> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
> because with the Vivante libraries, libEGL also needs libGAL. Usually,
> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
> this wasn't possible of course. We patched the gyp scripts to link
> against this libraries during building instead. I attached this patch.
> It is really a hack, because it circumvents the sandboxing process (this
> is why Chromium loads EGL and GLESv2 with dlopen() ).
>

Mahyar updated these patches to apply against the chromium-35.0.1883.0
build currently in meta-browser.

Additional notes to follow, but this appears to achieve HTML5 video
against Webm/Ogg videos when chromium is started with these command
line arguments:
	 --ignore-gpu-blacklist --enable-gpu --usegl-egl

Regards,


Eric
Carlos Rafael Giani - April 2, 2014, 10:21 a.m.
Keep in mind what I wrote. This version of VPU acceleration is very 
basic (it will copy frames with the CPU), and fulfilled the customer's 
immediate needs back then, but can be done much better. I am currently 
looking into a better approach.

On 2014-04-01 21:22, Eric Nelson wrote:
> Thanks again Carlos,
>
> On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
> >
> > <snip>
>>
>> Back then we needed some hw decoding quickly, so we did not look further
>> into this, since we had spent a lot of time getting the 2D acceleration
>> stable already. I could only briefly look at the exynos accelerator
>> code, but it does look like the right direction, although the VPU uses
>> physical addresses directly instead of dmabuf FDs. I am currently
>> writing a frontend for the libfslvpuwrap, which takes care of certain
>> complexities (like being able to pass userdata pointers to frames, to
>> make it possible to associate input and output frames, which is
>> essential for correct timestamping, especially when things like h264
>> frame reordering are active). This frontend will hide these things in a
>> future gstreamer-imx release to make the decoder code clearer and easier
>> to maintain, and is intended to be reusable, for example for Chromium
>> integration, or XBMC. Beyond that, my patches are probably not that
>> helpful anymore, since they were based on the vpx decoder way of
>> decoding, and I agree that the exynos code is a better starting point.
>> Nevertheless, I attached the patch. Keep in mind that it is very basic,
>> old, and wasn't further developed on because it was "good enough" for
>> the project.
>>
>> But here are some additional notes from our efforts to accelerate
>> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>>
>> 1. We started the google-chrome binary with these switches:
>> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
>> --enable-accelerated-2d-canvas
>>
>> 2. To make building Chromium easier, we switched to component build (by
>> default, it is all linked into one enormous binary). I attached a patch
>> that was necessary to fix some minor issues. Perhaps it is not necessary
>> anymore.
>>
>> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
>> because with the Vivante libraries, libEGL also needs libGAL. Usually,
>> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
>> this wasn't possible of course. We patched the gyp scripts to link
>> against this libraries during building instead. I attached this patch.
>> It is really a hack, because it circumvents the sandboxing process (this
>> is why Chromium loads EGL and GLESv2 with dlopen() ).
>>
>
> Mahyar updated these patches to apply against the chromium-35.0.1883.0
> build currently in meta-browser.
>
> Additional notes to follow, but this appears to achieve HTML5 video
> against Webm/Ogg videos when chromium is started with these command
> line arguments:
>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> Regards,
>
>
> Eric
>
Carlos Rafael Giani - April 2, 2014, 10:23 a.m.
On 2014-04-02 12:21, Boszormenyi Zoltan wrote:
> Hi,
>
> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.
>>
>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> you meant "--use-gl=egl".
>
> I have rebuilt Chromium with these patches. The result cannot play a 
> short MP4 video from a file:// URL.
> The console output is very suspicious, libGAL complains about a 
> missing ID file that actually exists.
> The image about the console is attached.
>
> Best regards,
> Zoltán Böszörményi
>

The VPU part could be because of missing firmware. Check if the vpu 
files are present in /lib/firmware.
As for MP4, this is a known problem. You are building Chromium, not 
Chrome. MP4 support is part of the restricted feature set, which is 
included in Chrome but not Chromium. Try a WebM file for example.
Gary Thomas - April 2, 2014, 10:28 a.m.
On 2014-04-02 04:21, Carlos Rafael Giani wrote:
> Keep in mind what I wrote. This version of VPU acceleration is very basic (it will copy frames with the CPU), and fulfilled the customer's immediate needs back then, but can be
> done much better. I am currently looking into a better approach.
> 
> On 2014-04-01 21:22, Eric Nelson wrote:
>> Thanks again Carlos,
>>
>> On 03/20/2014 04:19 PM, Carlos Rafael Giani wrote:
>> >
>> > <snip>
>>>
>>> Back then we needed some hw decoding quickly, so we did not look further
>>> into this, since we had spent a lot of time getting the 2D acceleration
>>> stable already. I could only briefly look at the exynos accelerator
>>> code, but it does look like the right direction, although the VPU uses
>>> physical addresses directly instead of dmabuf FDs. I am currently
>>> writing a frontend for the libfslvpuwrap, which takes care of certain
>>> complexities (like being able to pass userdata pointers to frames, to
>>> make it possible to associate input and output frames, which is
>>> essential for correct timestamping, especially when things like h264
>>> frame reordering are active). This frontend will hide these things in a
>>> future gstreamer-imx release to make the decoder code clearer and easier
>>> to maintain, and is intended to be reusable, for example for Chromium
>>> integration, or XBMC. Beyond that, my patches are probably not that
>>> helpful anymore, since they were based on the vpx decoder way of
>>> decoding, and I agree that the exynos code is a better starting point.
>>> Nevertheless, I attached the patch. Keep in mind that it is very basic,
>>> old, and wasn't further developed on because it was "good enough" for
>>> the project.
>>>
>>> But here are some additional notes from our efforts to accelerate
>>> Chromium on imx6 (keep in mind these apply to version 32.0.1664.3 ) :
>>>
>>> 1. We started the google-chrome binary with these switches:
>>> --ignore-gpu-blacklist --enable-gpu --use-gl=egl
>>> --enable-accelerated-2d-canvas
>>>
>>> 2. To make building Chromium easier, we switched to component build (by
>>> default, it is all linked into one enormous binary). I attached a patch
>>> that was necessary to fix some minor issues. Perhaps it is not necessary
>>> anymore.
>>>
>>> 3. Chromium tried to load libEGL with dlopen() , which caused problems,
>>> because with the Vivante libraries, libEGL also needs libGAL. Usually,
>>> build scripts just add -lEGL -lGAL to the linker line. With dlopen()
>>> this wasn't possible of course. We patched the gyp scripts to link
>>> against this libraries during building instead. I attached this patch.
>>> It is really a hack, because it circumvents the sandboxing process (this
>>> is why Chromium loads EGL and GLESv2 with dlopen() ).
>>>
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.

Are these patches available somewhere?

>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
Boszormenyi Zoltan - April 2, 2014, 10:29 a.m.
Hi,

2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>
> Mahyar updated these patches to apply against the chromium-35.0.1883.0
> build currently in meta-browser.
>
> Additional notes to follow, but this appears to achieve HTML5 video
> against Webm/Ogg videos when chromium is started with these command
> line arguments:
>      --ignore-gpu-blacklist --enable-gpu --usegl-egl

you meant "--use-gl=egl".

I have rebuilt Chromium with these patches. The result cannot play a short MP4 video from 
a file:// URL.
The console output is very suspicious, libGAL complains about a missing ID file that 
actually exists.
The image about the console is attached.

This is a copy of the original message, I cancelled waiting for moderation.
The original image exceeded the list limits so I reduced it to 16-color greyscale.

Best regards,
Zoltán Böszörményi
Carlos Rafael Giani - April 2, 2014, 10:33 a.m.
On 2014-04-02 12:28, Gary Thomas wrote:
> On 2014-04-02 04:21, Carlos Rafael Giani wrote:
>> Keep in mind what I wrote. This version of VPU acceleration is very basic (it will copy frames with the CPU), and fulfilled the customer's immediate needs back then, but can be
>> done much better. I am currently looking into a better approach.
> Are these patches available somewhere?

No, not yet, there is no finished code at the moment. If I am right, the 
necessary changes are not big though. The actual bottleneck here is time :)
Boszormenyi Zoltan - April 2, 2014, 11:12 a.m.
2014-04-02 12:23 keltezéssel, Carlos Rafael Giani írta:
> On 2014-04-02 12:21, Boszormenyi Zoltan wrote:
>> Hi,
>>
>> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>>
>>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>>> build currently in meta-browser.
>>>
>>> Additional notes to follow, but this appears to achieve HTML5 video
>>> against Webm/Ogg videos when chromium is started with these command
>>> line arguments:
>>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>>
>> you meant "--use-gl=egl".
>>
>> I have rebuilt Chromium with these patches. The result cannot play a short MP4 video 
>> from a file:// URL.
>> The console output is very suspicious, libGAL complains about a missing ID file that 
>> actually exists.
>> The image about the console is attached.
>>
>> Best regards,
>> Zoltán Böszörményi
>>
>
> The VPU part could be because of missing firmware. Check if the vpu files are present in 
> /lib/firmware.

I have /lib/firmware/vpu/vpu_fw_imxq6.bin, size 253968, the last 6 digits of md5sum is e8debf.

> As for MP4, this is a known problem. You are building Chromium, not Chrome. MP4 support 
> is part of the restricted feature set, which is included in Chrome but not Chromium. Try 
> a WebM file for example.

I tried these:

1. http://www.webmfiles.org/demo-files/
The webm examples play nicely, even scaled to full screen.

2. http://devfiles.myopera.com/articles/1891/custom-controls-webm-720p.html
It's the same "Elephants Dream" as one of the examples in (1) but at 720p.
It pays nicely in the browser control but it's a slideshow as full screen.

On the other hand, gstreamer 0.10 with gst-fsl-plugin and its gst-fsl-plugin-gplay
subpackage, gplay plays a 720p and a 1080p H.264 test file very well without tearing.

Best regards,
Zoltán Böszörményi
Christian Betz - April 2, 2014, 12:02 p.m.
>
> The VPU part could be because of missing firmware. Check if the vpu files
>> are present in /lib/firmware.
>>
> As for MP4, this is a known problem. You are building Chromium, not
> Chrome. MP4 support is part of the restricted feature set, which is
> included in Chrome but not Chromium. Try a WebM file for example.
>

this apparently can be worked around with gyp options:

"proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow
Chrome to play h.264 content, which is the only codec VAVDA knows about
today."

this is described on a wiki page setting up hw video decode on **intel**
processors:

https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode

note: i haven't actually tried this! (but i would like to)
Carlos Rafael Giani - April 2, 2014, 12:28 p.m.
On 2014-04-02 14:02, Christian Betz wrote:
>
>         The VPU part could be because of missing firmware. Check if
>         the vpu files are present in /lib/firmware.
>
>     As for MP4, this is a known problem. You are building Chromium,
>     not Chrome. MP4 support is part of the restricted feature set,
>     which is included in Chrome but not Chromium. Try a WebM file for
>     example.
>
>
> this apparently can be worked around with gyp options:
>
> "proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow 
> Chrome to play h.264 content, which is the only codec VAVDA knows 
> about today."
>
> this is described on a wiki page setting up hw video decode on 
> **intel** processors:
>
> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode
>
> note: i haven't actually tried this! (but i would like to)

We tried that back then. The Chrome branding enabled a million other 
things , which caused all sorts of difficulties. Also note that enabling 
the Chrome branding might have legal repercussions.
Eric Nelson - April 2, 2014, 2:16 p.m.
Hi Zoltan,

On 04/02/2014 03:29 AM, Boszormenyi Zoltan wrote:
> Hi,
>
> 2014-04-01 21:22 keltezéssel, Eric Nelson írta:
>>
>> Mahyar updated these patches to apply against the chromium-35.0.1883.0
>> build currently in meta-browser.
>>
>> Additional notes to follow, but this appears to achieve HTML5 video
>> against Webm/Ogg videos when chromium is started with these command
>> line arguments:
>>      --ignore-gpu-blacklist --enable-gpu --usegl-egl
>
> you meant "--use-gl=egl".
>

Probably right.

> I have rebuilt Chromium with these patches. The result cannot play a
> short MP4 video from a file:// URL.
> The console output is very suspicious, libGAL complains about a missing
> ID file that actually exists.
> The image about the console is attached.
>
> This is a copy of the original message, I cancelled waiting for moderation.
> The original image exceeded the list limits so I reduced it to 16-color
> greyscale.
>

There's a lesson here that you should be putting console output
either directly into your e-mail (trimmed to important points).
Putting the output into an image makes it slow to read and tedious
to quote.

 From what I can tell, you aren't running on an image that otherwise
supports GPU or VPU acceleration. That's a pre-requisite for this,
and I recommend using fsl-image-gui or some other known good
starting point.

Also note that we sent these patches not to imply that they're
fully functional, but to allow others to have a working example
for exploring the code.

Regards,


Eric
Boszormenyi Zoltan - April 4, 2014, 9:28 a.m.
2014-04-02 14:28 keltezéssel, Carlos Rafael Giani írta:
> On 2014-04-02 14:02, Christian Betz wrote:
>>
>>         The VPU part could be because of missing firmware. Check if the vpu files are
>>         present in /lib/firmware.
>>
>>     As for MP4, this is a known problem. You are building Chromium, not Chrome. MP4
>>     support is part of the restricted feature set, which is included in Chrome but not
>>     Chromium. Try a WebM file for example.
>>
>>
>> this apparently can be worked around with gyp options:
>>
>> "proprietary_codecs=1 ffmpeg_branding=Chrome branding=Chrome to allow Chrome to play 
>> h.264 content, which is the only codec VAVDA knows about today."
>>
>> this is described on a wiki page setting up hw video decode on **intel** processors:
>>
>> https://code.google.com/p/chromium/wiki/LinuxHWVideoDecode
>>
>> note: i haven't actually tried this! (but i would like to)
>
> We tried that back then. The Chrome branding enabled a million other things , which 
> caused all sorts of difficulties. Also note that enabling the Chrome branding might have 
> legal repercussions.

A comment on that page says to use these flags (Chrome branding is not needed, only 
chromeos=1):

export GYP_DEFINES="chromeos=1 proprietary_codecs=1 ffmpeg_branding=Chrome"

do_configure succeeds but do_compile fails with "brlapi.h" missing. A google search
gave me this: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/StpTzFnf8ok
Supposedly, running "build/install-build-deps.sh" fixes it but it turned out it only fixes
things for building Chromium for the host and the script wants an Ubuntu host OS.

Is there a ready to use recipe for brlapi/brltty somewhere or should I make one?

Thanks in advance,
Zoltán Böszörményi

Patch

From f059d6501b17ab8aed618f53f7e4255cf8d8326b Mon Sep 17 00:00:00 2001
From: Mahyar Yaghmaee <mahyar@boundarydevices.com>
Date: Fri, 28 Mar 2014 14:06:40 -0700
Subject: [PATCH 3/3] Update vpu_video_decoder.cc to match chromium-35.0.1883

Signed-off-by: Mahyar Yaghmaee <mahyar@boundarydevices.com>
---
 media/filters/vpu_video_decoder.cc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/media/filters/vpu_video_decoder.cc b/media/filters/vpu_video_decoder.cc
index 396d4f1..9024484 100644
--- a/media/filters/vpu_video_decoder.cc
+++ b/media/filters/vpu_video_decoder.cc
@@ -18,7 +18,7 @@ 
 #include "base/message_loop/message_loop_proxy.h"
 #include "base/strings/string_number_conversions.h"
 #include "base/sys_byteorder.h"
-#include "media/base/bind_to_loop.h"
+#include "media/base/bind_to_current_loop.h"
 #include "media/base/decoder_buffer.h"
 #include "media/base/demuxer_stream.h"
 #include "media/base/media_switches.h"
@@ -393,7 +393,7 @@  void VpuVideoDecoder::Decode(const scoped_refptr<DecoderBuffer>& buffer,
 
   // Return empty frames if decoding has finished.
   if (state_ == kDecodeFinished) {
-    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEOSFrame());
     return;
   }
 
@@ -445,7 +445,7 @@  void VpuVideoDecoder::DecodeBuffer(const scoped_refptr<DecoderBuffer>& buffer) {
   // Transition to kDecodeFinished on the first end of stream buffer.
   if (state_ == kNormal && buffer->end_of_stream()) {
     state_ = kDecodeFinished;
-    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEmptyFrame());
+    base::ResetAndReturn(&decode_cb_).Run(kOk, VideoFrame::CreateEOSFrame());
     return;
   }
 
-- 
1.8.3.2