From patchwork Tue Aug 6 09:05:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonas Karlman X-Patchwork-Id: 35165 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:612c:1f5a:b0:489:2eb3:e4c4 with SMTP id jm26csp1907236vqb; Tue, 6 Aug 2024 02:06:40 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCU1MeUnDoFpbRiyntrBtc48scdotWPIIoXNRNudBjrSNZo7pZmjj0nGRaQ9+BEkkopVxlP4CZp9vW2QyTceIZhEoy8l3kAKteiurg== X-Google-Smtp-Source: AGHT+IEtkgb/7rCwGFGfyvnuq6yp4ox7ZG3LMXuiZhaLWzK24glo2dcnIPAWs8xuNFDtkZ7vq7mA X-Received: by 2002:aa7:d309:0:b0:5bb:9ae0:4a3e with SMTP id 4fb4d7f45d1cf-5bb9ae04b08mr447455a12.29.1722935199764; Tue, 06 Aug 2024 02:06:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1722935199; cv=none; d=google.com; s=arc-20160816; b=me0qymhYK3DP7WWgTFaBNPZ1jQfV45IBGJbY7+sAsAQe8TJCUopuixacL+vqefncMP z8a3BhNSNYVG+Q0aNk558Q3K/Pwq0BrE6IwdnzfqRYkn6ywS8natcA1YthYJ40Fxv31b l2Ro0QsiP1muFx+8I7QmQk9vAKiZSOOqTjaDrIRvtjaTE9IgPtPPcC8YvXMW9Q+UeBiM oKKN0aZhMX4ikKnoyzA6diZiGbxaBrtwJKYRFmu087ABByT8RPwbpT/4YeIE0LOnVkD0 b6SzxuRixc5iDFyRU8w9PBDf3+Ku6ImRx+J0ThmoxYajI03d7r/JoO+spmEX2HPsiQpM pJVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=sjbq/XpWlRhPSxcnndhWzqkH/e9Lby8Y9qqQkGQeXJU=; fh=F0y82oXdVu1PBU0SjSpsDDU9xEVfmDCmSjHZdwqvnK8=; b=oiYTd/L94Mn6xMh1OLaYFdBcF4DZqqmONxEQT3PZodRb/F1xAFCMfMY5873SIOR0kw nZ/7yJCJaf1iojSoWGjoj8Q/bsAo5lB+yE7I+U1bRrmPb5oaRM0x0dFI5Mm/6M8OPqU+ U3Pa9EbJDoKSDOJmJ6AtbTJAfKA7IA3iuheDzO5GbLiJttcuh7tjBvLg8eDS2dRpX6cm tBFpVhBekJDcGSljbzN6sSLBzA2puUPnFpgkHn/OwL7uVYQP0EC2fhDSsPFAB6C/Vvp8 YDfjDxhsdA6adqrux28+6r/9X5Sndx+G8BbhxlhluCEC+Quqiz3u5R1LW7IicQy1CCs1 NvYQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@kwiboo.se header.s=fe-e1b5cab7be header.b=zywT5HQN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kwiboo.se Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-5b83b934e56si5684819a12.318.2024.08.06.02.06.39; Tue, 06 Aug 2024 02:06:39 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kwiboo.se header.s=fe-e1b5cab7be header.b=zywT5HQN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kwiboo.se Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B8AF068D94D; Tue, 6 Aug 2024 12:06:34 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from smtp.forwardemail.net (smtp.forwardemail.net [164.92.70.200]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3201868D5FE for ; Tue, 6 Aug 2024 12:06:26 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kwiboo.se; h=Content-Transfer-Encoding: MIME-Version: Message-ID: Date: Subject: Cc: To: From; q=dns/txt; s=fe-e1b5cab7be; t=1722935181; bh=60I57r8MnjL1suHnBOqwRdF8PPJhnuMhNXFSVda96OE=; b=zywT5HQNtH//E8USYrpDU5++0XCjU6bwDLqIkGlt59niiQr0Dzcjk5oBqEjND/JvjpnJLPlOT /KklU2hO9oX+nVp+sR452QrAgFGV9YGrNyky5+2IDi98wNjPs8kntLTQ9prv2pws3GLd3V3uKBq upKrGCTmboShoiQvFb8FjhLHWtbPGRKQmS6HUYJ8rE6S0mAgUwkRqK9eh02fetK7hh+JKAXOIyM N67NVdAWunbYcoqZ9mRs5U25UAgZF3MK39SvFLm8Fl/UcEURcLnl+5cbbtB8CqT40P2KE/Vm7rB 4DQea13hNcaUPvYBJXOQXItyKfJZkRYEJoHfUnHu232A== From: Jonas Karlman To: ffmpeg-devel@ffmpeg.org Date: Tue, 6 Aug 2024 09:05:59 +0000 Message-ID: <20240806090607.43240-1-jonas@kwiboo.se> X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 X-Report-Abuse-To: abuse@forwardemail.net X-Report-Abuse: abuse@forwardemail.net X-Complaints-To: abuse@forwardemail.net X-ForwardEmail-Version: 0.4.40 X-ForwardEmail-Sender: rfc822; jonas@kwiboo.se, smtp.forwardemail.net, 164.92.70.200 X-ForwardEmail-ID: 66b1e78c8ac7bd7d98d34c28 Subject: [FFmpeg-devel] [PATCH v2 0/8] Add V4L2 Request API hwaccels for MPEG2, H.264 and HEVC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Benjamin Gaignard , Jonas Karlman , Alex Bee , Jernej Skrabec , Boris Brezillon , Nicolas Dufresne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: t3MIpO6XfulW This is a follow up to a very old series from April 2019 [1] and December 2020 [2], adding V4L2 Request API hwaccels for stateless decoding of MPEG2, H.264 and HEVC. These hwaccels has in one form or another been used in LibreELEC community, nightly and release images since Dec 20th 2018. HISTORY The initial v4l2-request hwaccel code was mainly created as a proof of concept at the end of 2018 by me and Jernej Skrabec. Back when Bootlin's crowdfunding campaign for upstream Linux kernel driver for Allwinner VPU had started to bear fruit. At the time we had very little knowledge on how to properly interact with any of the V4L2 APIs, and the existing V4L2 M2M code in FFmpeg seemed very complex to start working with, so we started from scratch. (That is the main reason why these hwaccels still does not use any of the existing V4L2 M2M code in FFmpeg.) The hwaccels had one major limitation, it waited on the kernel to complete decoding before continuing with next request. Due to hwaccels still was able to decode up to 4k 50-60fps on Allwinner and Rockchip boards, time was never spent trying to improve this limitation. The initial version of these hwaccels was merged as patches into LibreELEC in April 2019 [3], with my FFmpeg tree as main source for any update of FFmpeg v4l2-request patches included in LibreELEC. A RFC was sent in April 2019 [1], with very little feedback and the required kernel headers not being merged until a year later there was never any new RFC revision submitted. After the release of the Raspberry Pi 4 in June 2019 there was hope that RPi folks would rework and/or improve our proof-of-concept to something better that could be upstreamed, as history has show this did not happen. And instead we have just ended up with a second FFmpeg v4l2-request implementation that has only really been used for HEVC decoding on RPi. Due to personal reasons I took a long break from doing open-source contributions during ~2020-2023. During this time Jernej's FFmpeg tree became the main source for FFmpeg v4l2-request patches in LibreELEC [4]. In December 2020 a v1 of this series was sent [2], at that time kernel headers for H.264 had been merged into Linux kernel, and I cannot remember why it never went any further. Last update to the v4l2-request patches in LibreELEC was done by me in November 2023 [5]. Along the way there has been noteworthy contributions including from Boris Brezillon, Ezequiel Garcia, Alex Bee and Benjamin Gaignard. PRESENT DAY The version submitted in this series have seen major refactoring to the common code, to make it more ready for upstreaming and also a rework of how buffers and requests are handled. The limitation of waiting for decoding to complete has been removed and now multiple pending requests can be in-flight at the same time. This was a requirement to better support HEVC decoding on RPi. The older versions also incorrectly required use of a DRM hwdevice, however V4L2 decoding has nothing to do with the DRM subsystem in Linux. Instead in this version a new V4L2 Request API hwdevice has been added. Thanks to this it is now possible to specify what media device to use for decoding, in case multiple decoders exists on a system. E.g. using a -init_hw_device v4l2request:/dev/media1 parameter. This version only add support for MPEG2, H264 and HEVC. Support for VP8, VP9 and AV1 is planned and be added in next revision or in a follow up series. HOW TO USE To use the V4L2 Request API hwaccels you must build FFmpeg on a system with recent Linux kernel headers, v6.0+. It also requires libdrm and libudev for the hwaccels to be enabled and successfully build. This can then be runtime tested on multiple Allwinner and Rockchip devices. To runtime test this on a RPi 4 or 5 you should use latest rpi-6.6.y kernel. ffmpeg -hwaccel v4l2request -hwaccel_output_format drm_prime \ -i -map 0:v -f null - This series has been tested with cedrus driver on Allwinner H6, hantro and rkvdec driver on Rockchip RK3399, and rpivid driver on RPi 4/5. PRs have also been opened for Kodi and mpv to assist with the transition from using a DRM hwdevice type to a new V4L2REQUEST hwdevice type. - Kodi: https://github.com/xbmc/xbmc/pull/25467 - mpv: https://github.com/mpv-player/mpv/pull/14511 With those PRs applied it should be possible to playback video using kodi-gbm or mpv, see the PRs above for more details. It should also be possible to run fluster test suites with following PR: - fluster: https://github.com/fluendo/fluster/pull/179 FUTURE I am expecting that there will be a new revision of this series adding VP8, VP9 and AV1 support in a week or two. Until then, please get back with any type of feedback. A copy of this series can also be found at [6]. [1] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2019-April/242316.html [2] https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-December/273579.html [3] https://github.com/LibreELEC/LibreELEC.tv/pull/3405 [4] https://github.com/LibreELEC/LibreELEC.tv/commits/master/packages/multimedia/ffmpeg/patches/v4l2-request [5] https://github.com/LibreELEC/LibreELEC.tv/pull/8356 [6] https://github.com/Kwiboo/FFmpeg/commits/v4l2request-2024-v2/ Boris Brezillon (1): avcodec/h264dec: add ref_pic_marking and pic_order_cnt bit_size to slice context Jernej Skrabec (2): avcodec: Add V4L2 Request API h264 hwaccel avcodec: Add V4L2 Request API hevc hwaccel Jonas Karlman (5): avutil/hwcontext: Add hwdevice type for V4L2 Request API avcodec: Add common V4L2 Request API code avcodec/v4l2request: Probe for a capable media and video device avcodec/v4l2request: Add common decode support for hwaccels avcodec: Add V4L2 Request API mpeg2 hwaccel MAINTAINERS | 1 + configure | 21 + libavcodec/Makefile | 4 + libavcodec/h264_slice.c | 13 +- libavcodec/h264dec.c | 3 + libavcodec/h264dec.h | 2 + libavcodec/hevc/hevcdec.c | 10 + libavcodec/hwaccels.h | 3 + libavcodec/hwconfig.h | 2 + libavcodec/mpeg12dec.c | 6 + libavcodec/v4l2_request.c | 452 +++++++++++++++++ libavcodec/v4l2_request.h | 107 ++++ libavcodec/v4l2_request_decode.c | 459 +++++++++++++++++ libavcodec/v4l2_request_h264.c | 523 +++++++++++++++++++ libavcodec/v4l2_request_hevc.c | 790 +++++++++++++++++++++++++++++ libavcodec/v4l2_request_internal.h | 51 ++ libavcodec/v4l2_request_mpeg2.c | 176 +++++++ libavcodec/v4l2_request_probe.c | 614 ++++++++++++++++++++++ libavutil/Makefile | 3 + libavutil/hwcontext.c | 4 + libavutil/hwcontext.h | 1 + libavutil/hwcontext_internal.h | 1 + libavutil/hwcontext_v4l2request.c | 261 ++++++++++ libavutil/hwcontext_v4l2request.h | 41 ++ 24 files changed, 3547 insertions(+), 1 deletion(-) create mode 100644 libavcodec/v4l2_request.c create mode 100644 libavcodec/v4l2_request.h create mode 100644 libavcodec/v4l2_request_decode.c create mode 100644 libavcodec/v4l2_request_h264.c create mode 100644 libavcodec/v4l2_request_hevc.c create mode 100644 libavcodec/v4l2_request_internal.h create mode 100644 libavcodec/v4l2_request_mpeg2.c create mode 100644 libavcodec/v4l2_request_probe.c create mode 100644 libavutil/hwcontext_v4l2request.c create mode 100644 libavutil/hwcontext_v4l2request.h