With dmabuf support already implemented it should be feasible to implement VAAPI playback by ffmpeg.

I already checked that the playback works fine on Wayland/Intel on mpv player which also uses ffmpeg to decode frames and vaExportSurfaceHandle() to convert AVFrame to dmabuf memory layout.

Some integration work needs to be done to carry dmabuf buffer by MediaDataDecoder::DecodedData to correct TextureClient to map it as EGL texture.

playback works fine on Wayland/Intel on mpv player

This require intel-vaapi-driver 2.4.0+, correct? At least according to https://github.com/intel/intel-vaapi-driver/issues/449

I use Fedora 31 as a reference distro with latest updates (libva-2.6.0).

Do you also plan to implement video encoding hardware acceleration through VAAPI? It could help quite a lot for video conferencing scenarios.

And a bit of a side question, in case of Vulkan + Wayland, would it use something else rather than EGL?

(In reply to Shmerl from comment #3)

Do you also plan to implement video encoding hardware acceleration through VAAPI? It could help quite a lot for video conferencing scenarios.

No, that's implemented by WebRTC project (https://webrtc.org/) so it needs to be implemented there. This work also does not help with video played via WebRTC like video conferencing.

You mean that Firefox uses WebRTC project code, and I should check there for VAAPI support?

WebRTC on Linux a major problem for clients like Cisco Webex, which constantly chokes in Firefox, complaining that "CPU is overutilized". I thought may be hardware acceleration can help it.

(In reply to Shmerl from comment #6)

You mean that Firefox uses WebRTC project code, and I should check there for VAAPI support?

Yes, exactly. Firefox uses WebRTC for video conferences which also includes video encode/decode there.

Hm, according to https://webrtc.org/web-apis/firefox/

For specific issues with the FIrefox WebRTC code, please use our bug tracker

And it points back to bugzilla here. I can open another bug, for WebRTC component.

(In reply to Shmerl from comment #4)

And a bit of a side question, in case of Vulkan + Wayland, would it use something else rather than EGL?

It will use vulkan extentions instead of EGL.
Importing VAAPI decoded frames to vulkan surface is not yet supported. There is a similar concept implemented on mpv though which fails on my configuration(amdgpu).

Pardon me if this is off-topic, but is this bug tracking all implementations of VA-API, or just the Intel one for now?

(In reply to Sandi Vujaković from comment #10)

Pardon me if this is off-topic, but is this bug tracking all implementations of VA-API, or just the Intel one for now?

This bug is tracking a playback by ffmpeg (image decode) and libva (decoded image transfer to dmabuf).

Just for the reference, if anyone will be looking. I opened WebRTC bug for VAAPI support: https://bugs.chromium.org/p/webrtc/issues/detail?id=11296

(In reply to Martin Stránský [:stransky] from comment #7)

Yes, exactly. Firefox uses WebRTC for video conferences which also includes video encode/decode there.

Really unfortunate that we're using the reference/Chrome implementation (libwebrtc). In GStreamer, this would be easy to do >_<

But anyway, if anyone would be interested in modifying libwebrtc to do this, seems like there was an attempt to do a similar thing for Android: bug 969395.

Is Firefox on Android not using libwebrtc? Otherwise it's upstream issue like above.

WIP patch. Video has wrong colors (perhaps due to wrong YUV color profile) and frames are shuttered. But decodes frames on gpu and pass them by dmabuf to compositor as gl textures.

(In reply to Martin Stránský [:stransky] from comment #15)

Created attachment 9122629 [details] [diff] [review]
WIP

WIP patch. Video has wrong colors (perhaps due to wrong YUV color profile) and frames are shuttered. But decodes frames on gpu and pass them by dmabuf to compositor as gl textures.

I remember having problems like this when trying to get VAAPI/DRI3 to work with Chromium/X11 under Wayland. It worked fine with Weston, but under GNOME it would have the wrong colour profile, due to necessary colourspace conversions not being fully supported in mutter. I don't know if the situation is still the same, but I wouldn't be all that surprised if it exercised similar code paths with wayland-egl-dmabuf. Maybe this is a Red Herring, but I thought I'd mention it.

Thanks, I already solved the wrong colors with moving to WebRender which supports NV12 format correctly. Last issue remaining is frame scattering, seems to be caused by conversion from DRI prime to GL texture - at least ffmpeg internal conversion works fine.

Is this hooked up to the "Media Capabilities API" so powerEfficient return true if VAAPI is available?

(In reply to Kristian Klausen from comment #19)

Is this hooked up to the "Media Capabilities API" so powerEfficient return true if VAAPI is available?

This doesn't exist on any OS - see bug 1569686 to track this.

Martin,
In the WIP patch you attached earlier, you call av_hwdevice_ctx_create(). It's possible to build ffmpeg without CONFIG_LIBDRM, and in that case it will call vaGetDisplay() and not vaGetDisplayDRM() expected by your code. To ensure X11 version is never called, you can specify "connection_type" in options:

bool FFmpegDataDecoder<LIBAV_VER>::CreateVAAPIDeviceContext() {
  AVDictionary *opts = nullptr;
  mLib->av_dict_set(&opts, "connection_type", "drm", 0);
  if (mLib->av_hwdevice_ctx_create(&mVAAPIDeviceContext, AV_HWDEVICE_TYPE_VAAPI,
                                   NULL, opts, 0) < 0) {
    mLib->av_dict_free(&opts);
    …

Thanks, I'll update the patch.

Hi, just a small question if I'm not bothering you. When will the initial patches land in nightly? I would like to participate with testing this, but my PC dislikes compiling from source because my CPU is too weak.

(In reply to Marko from comment #23)

Hi, just a small question if I'm not bothering you. When will the initial patches land in nightly? I would like to participate with testing this, but my PC dislikes compiling from source because my CPU is too weak.

I have no idea.

First version of the patch for latest trunk, seems to be working somehow.

All needed patches are submitted in the tracked bugs here so we'll have working VAAPI on Wayland when these ones hit mozilla-central.
My performance measures are here - https://bugzilla.mozilla.org/show_bug.cgi?id=1616185#c7

(In reply to Martin Stránský [:stransky] from comment #26)

All needed patches are submitted in the tracked bugs here so we'll have working VAAPI on Wayland when these ones hit mozilla-central.
My performance measures are here - https://bugzilla.mozilla.org/show_bug.cgi?id=1616185#c7

What preferences are needed to enable all of it?

(In reply to Martin Stránský [:stransky] from comment #26)

All needed patches are submitted in the tracked bugs here so we'll have working VAAPI on Wayland when these ones hit mozilla-central.
My performance measures are here - https://bugzilla.mozilla.org/show_bug.cgi?id=1616185#c7

Awesome! To bad there are no reaction possibilities in bugzilla, this post deserves a thousand thumbs up and the likes.

Looks like you could rightfully claim https://www.bountysource.com/issues/55506502-add-va-api-hardware-decoding-support-on-linux now :)

If you can document all the required preferences, I'll enable it for testing.

It doesn't work yet, Bug 1616129 and Bug 1616185 need to be fixed first.

As Bug 1616185 landed you should be able to run va-api accelerated video playback on Wayland with latest nightly. I just tried 20200302212732 build and it works for me on Fedora 31 / Gnome / Mesa DRI Intel(R) UHD Graphics 630.

In order to run it you need to:

• run Firefox under wayland compositor with MOZ_ENABLE_WAYLAND=1
• set widget.wayland-dmabuf-vaapi.enabled to true at about:config
• enable appropriate HW acceleration for Firefox (GL compositor or WebRender), verify that in about:support
• have libva installed (I have libva-2.6.0-0.1.fc31.x86_64 on Fedora)

You can verify that VA-API is enabled by running Firefox with MOZ_LOG="PlatformDecoderModule:5" env variable and check in the log output that VA-API is enabled and used.

Confirmed. vaapi doesn't work with gl either. Aidan too reported the same error.

I see the same thing on Arch Linux with Intel iGPU:

[Child 14174: Main Thread]: D/PlatformDecoderModule Couldn't load function avcodec_get_hw_config
[Child 14174: Main Thread]: D/PlatformDecoderModule Couldn't load function av_hwdevice_ctx_create
[Child 14174: Main Thread]: D/PlatformDecoderModule Couldn't load function av_hwframe_transfer_get_formats
[Child 14174: Main Thread]: D/PlatformDecoderModule Couldn't load function av_hwdevice_ctx_create_derived
[Child 14174: Main Thread]: D/PlatformDecoderModule Sandbox decoder rejects requested type

I verified that libva.so.2.600.0, libva-drm.so.2.600.0 and libva-x11.so.2.600.0 are loaded in the content process. I also attached gdb to the content process and checked that disas can find the four functions above.

(In reply to Akarshan Biswas from comment #32)

Doesn't work on amdgpu with webrender.

amdgpu should be supported, It worked for me when I tested it some time ago. If there's any bug I'll fix that.

from the log:
[Child 2692: MediaPDecoder #1]: D/PlatformDecoderModule VA-API FFmpeg is disabled by platform

that means:

• you're not running on Wayland or dmabuf is not supported
• it's disabled by a preference or GL/Webrender is disabled

Please file a new bug for it and attach about:support page and also firefox log obtained by running:

MOZ_ENABLE_WAYLAND=1 MOZ_LOG="WidgetWayland:5" ./firefox

Thanks.

Guys, please file a new bug for it against Core / Audio,Video playback component (or just clone this one) and drop a link here.
Thanks.

(In reply to Martin Stránský [:stransky] from comment #35)

[Child 2692: MediaPDecoder #1]: D/PlatformDecoderModule VA-API FFmpeg is disabled by platform
I don't think the error is related to this. The error is on bug id 1616128 I think (loading library symbols from ffmpeg libraries )

But anyway, I will open a new bug report with all the logs.

Video flickering on BBC video player when controls appear and disappear:
https://bugzilla.mozilla.org/show_bug.cgi?id=1621951

https://crash-stats.mozilla.org/report/index/87c543ef-efc7-4da8-abeb-243a70200318

This crash happens to me on Gnome shell 3.36.0 / Mutter 3.36.0 on Fedora 32 beta. It worked on Gnome Shell 3.34.3 / Mutter 3.34.3,

Is this bug already documented?

Hm why is it marked as fixed if it still occurs?

It was fixed 2 hours ago. Your build id is 20200318041411 (2020-03-18 04:14:11). Now we have to wait for the next update of Nightly.
Edit: You could also try out Linux x64 shippable opt build from bug 1622729 comment 4. If that crash still occurs, please comment in bug 1622729.

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #42)

It was fixed 2 hours ago. Your build id is 20200318041411 (2020-03-18 04:14:11). Now we have to wait for the next update of Nightly.
Edit: You could also try out https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/A_UE9lP6T4aNqonpQDe-eQ/runs/0/artifacts/public/build/target.tar.bz2 (Linux x64 shippable opt). If it doesn't work, please comment in bug 1622729, not here.

Ooooooh such a coincidence that it got fixed just today :D I've been eyeing this bug for days now.

https://i.imgur.com/1jxhIe1.png

Why doesn't it work for 4k 60fps videos on youtube yet? The CPU usage goes bonkers and videos stutter like crazy. I have intel UHD 620 and my vainfo shows all H264, VP9 and HEVC decoding supported.

You might not be using VAAPI at all because of the sandbox (bug 1619585). Stuttering especially in case of overlays is known (bug 1619882) - or you might be decoding on the CPU. If you have multiple GPUs, it's bug 1622132. Only use https://nightly.mozilla.org for VAAPI testing.
Run $ GDK_BACKEND=wayland MOZ_LOG="PlatformDecoderModule:5" path/to/firefox to find out what's going on. Please create an own ticket and attach the log to it if you can't figure out what's the cause. If it's the sandbox, try to use env var LIBVA_DRIVER_NAME=i965 as well.

(In reply to Marko from comment #47)

https://gitlab.freedesktop.org/snippets/922
Sry here is the proper link for the log.

Marko, please open individual issues. This is a general bug for the implementation with lots of people subscribed that are not necessarily interested why things don't work on individual setups. You can then mark your new bugs as blockers for this one. Thx.

I wander if Bug 869813 "Option to set WebM or H.264 as preferred" could help Bug 1625431 and/or Bug 1624103
Not sure.
Thanks for your work.

Did anyone notice weird stuttering/shaking issues with 60 FPS H264 videos? They seem to happen exactly when the Youtube control appear and disappear. I'm on Arch Linux with Firefox 75 Wayland and intel-media-driver (LIBVA_DRIVER_NAME=iHD), with Intel UHD Graphics 630 integrated graphic card. No problems at all with 30 fps videos. Mpv seem to work fine with vaapi and 60 FPS H264 videos.

Just for the reference, since libwebrtc support came up before, I commented here with some details from libwebrtc developers: https://bugzilla.mozilla.org/show_bug.cgi?id=969395#c101

who basically recommend for Firefox to implement VAAPI support outside of libwebrtc anyway. So may be this effort can be somehow shared with the one for regular video handling.

I just updated nightly and while now dmabuf.video-textures is not needed, I get some green frames when GPU decoding. I don't see it mentioned somewhere which is why I comment. Maybe related to #1645673 ?

With Bug 1619523 added (vaapi via dmabuf/x11) let's track all linux va-api issues as the X11/Wayland va-api&dmabuf code is almost identical.

(In reply to gsf.greece from comment #53)

I just updated nightly and while now dmabuf.video-textures is not needed, I get some green frames when GPU decoding. I don't see it mentioned somewhere which is why I comment. Maybe related to #1645673 ?

Let's finish X11/dmabuf/vaapi implementation first. I'll look at those minor ones after that as I expect they affect both X11 and Wayland backends.

Martin, are you going to make vaapi on X11 too? I understood that you could not and you left this work to other person. Or are you helping him? It's curiosity to know if this work is going to some place by you or by other that I have not seen XD.

And about the post in the other site, okey, didn't know it. But nevermind, thank to you for this.

(In reply to albertogomezmarin from comment #56)

Martin, are you going to make vaapi on X11 too?

Yes, a basic implementation is waiting at Bug 1619523.

but anyone are(In reply to Martin Stránský [:stransky] from comment #57)

(In reply to albertogomezmarin from comment #56)

Martin, are you going to make vaapi on X11 too?

Yes, a basic implementation is waiting at Bug 1619523.

but anybody has yet be assigned to this work. so until anybody go ahead you cant do nothing. right?

I have tried EGL on my computer with amdgpu drivers. The only thing I noticed is that the close windows is transprent like in the nvidia driver bug. But I think is another thing in WR.

Apart from this, I have tried to use vaapi and it doesn't work. I have seen VCN block not being used when playing a video in youtube so..
The video was in VP9 format that my ryzen is capable to decode and I have tested in chromium so...
maybe someone can help me a little bit?

Firefox-nightly with EGL and dmabuff and vaapi enabled things in about:config

(In reply to albertogomezmarin from comment #59)

I have tried EGL on my computer with amdgpu drivers. The only thing I noticed is that the close windows is transprent like in the nvidia driver bug. But I think is another thing in WR.

Apart from this, I have tried to use vaapi and it doesn't work. I have seen VCN block not being used when playing a video in youtube so..
The video was in VP9 format that my ryzen is capable to decode and I have tested in chromium so...
maybe someone can help me a little bit?

Firefox-nightly with EGL and dmabuff and vaapi enabled things in about:config

For VP9 videos you'll currently also need to disable media.ffvpx.enabled.

For further debugging question please also refer to https://mastransky.wordpress.com/2020/06/03/firefox-on-fedora-finally-gets-va-api-on-wayland/ and try running with MOZ_LOG="PlatformDecoderModule:5" (explained in the blog post).

(In reply to Robert Mader [:rmader] from comment #60)

(In reply to albertogomezmarin from comment #59)

I have tried EGL on my computer with amdgpu drivers. The only thing I noticed is that the close windows is transprent like in the nvidia driver bug. But I think is another thing in WR.

Apart from this, I have tried to use vaapi and it doesn't work. I have seen VCN block not being used when playing a video in youtube so..
The video was in VP9 format that my ryzen is capable to decode and I have tested in chromium so...
maybe someone can help me a little bit?

Firefox-nightly with EGL and dmabuff and vaapi enabled things in about:config

For VP9 videos you'll currently also need to disable media.ffvpx.enabled.

For further debugging question please also refer to https://mastransky.wordpress.com/2020/06/03/firefox-on-fedora-finally-gets-va-api-on-wayland/ and try running with MOZ_LOG="PlatformDecoderModule:5" (explained in the blog post).

done. I have been testing and firefox works with vaapi in a x11 .
I have been testing and this was my benchmarking:
cpu usage:
firefox stable with webrender: 26%
firefox vaapi: 14%
chromium vaapi: 6%
all was done in a fast way with VP9 and 1080p and youtube

But at least we have at the end vaapi

Thanks Alberto. To really measure the effect you'd need to compare the same FF version with and without VAAPI however, as a lot of work goes into optimizing WR atm.

For further optimizations AFAIK it's worth to keep an eye on frame building (bug 1596042 - bug 1650378 sound especially relevant here) and partial damage (bug 1625070 etc.)

(In reply to Robert Mader [:rmader] from comment #62)

Thanks Alberto. To really measure the effect you'd need to compare the same FF version with and without VAAPI however, as a lot of work goes into optimizing WR atm.

For further optimizations AFAIK it's worth to keep an eye on frame building (bug 1596042 - bug 1650378 sound especially relevant here) and partial damage (bug 1625070 etc.)

I am on exams right now so I cant do more tests.
But in the future I will. I hope this can improve the laptop experience in linux. And I expect firefox to be more power efficience than chromium that is the goal with this I think. I will do power tests in my laptop in the future so you can have the numbers

On a Mac Mini 2009 with Nvidia onboard GPU NVIDIA Corporation C79 [GeForce 9400] (rev b1) with 256MB RAM and nouveau 1.0.16 drivers. I get accelerated (?) but corrupt video. Attaching the video screenshot. It looks similar to the wayland issue reported on the other bug.
I can reproduce logs etc but I am not sure if they are needed.

(In reply to ilgaz from comment #65)

On a Mac Mini 2009 with Nvidia onboard GPU NVIDIA Corporation C79 [GeForce 9400] (rev b1) with 256MB RAM and nouveau 1.0.16 drivers. I get accelerated (?) but corrupt video. Attaching the video screenshot. It looks similar to the wayland issue reported on the other bug.
I can reproduce logs etc but I am not sure if they are needed.

Please file a new bug for it.
Thanks.

I'm having trouble getting this working with proprietary nvidia-drivers on Linux with patched firefox 80 (vaapi fix backported from firefox 81). The log consistently complains about lack of webrender/dmabuf despite doing all the usual (webrender enabled, EGL enabled via environmental variable, relevant config options set).

I was going to find a bug, but I realized it might be not working because proprietary nvidia-drivers can't do dma-buf? There seems to be a lot of confusion about this around the net and I was hoping someone could clarify if it should be able to work so I can file a bug (or not).

(In reply to Gregory from comment #67)

I'm having trouble getting this working with proprietary nvidia-drivers on Linux with patched firefox 80 (vaapi fix backported from firefox 81). The log consistently complains about lack of webrender/dmabuf despite doing all the usual (webrender enabled, EGL enabled via environmental variable, relevant config options set).

I was going to find a bug, but I realized it might be not working because proprietary nvidia-drivers can't do dma-buf? There seems to be a lot of confusion about this around the net and I was hoping someone could clarify if it should be able to work so I can file a bug (or not).

Note this is on x11.

Proprietary Nvidia doesn't support GPL-licensed DMABUF Linux kernel API. There is no confusion. They would need to provide the required patches/firmware to the Mesa/Nouveau project. Please try to use your integrated graphics instead. bug 1658772 is about Mesa/Nouveau.

(In reply to Gregory from comment #67)

I was going to find a bug

There is already a bug about VDPAU/NVDEC support: bug 1210729.
VDPAU is not restricted to NVIDIA only, it's available for Gallium-based Mesa drivers too. But since that API is one of NVIDIA's native APIs, bug 1210729 is the most relevant tracking bug.

(In reply to Darkspirit, Servo QA from comment #69)

Proprietary Nvidia doesn't support GPL-licensed DMABUF Linux kernel API. There is no confusion.

Apparently there is for “normal” users. Could the logging be improved? Separating the message about DMABUF and WebRender for example, and explicitly mentioning the driver as a cause of errors in the logs?