From patchwork Fri Jul 1 20:48:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 34762 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp1261021pzh; Fri, 1 Jul 2022 13:49:16 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tCqX/8mCcSqvU70POk9cx7cDQ3uZHqqeRRDwGfzGHaPA0poUL+GywXD5IkNa3XxLPpkjv1 X-Received: by 2002:a17:907:7245:b0:726:a909:902e with SMTP id ds5-20020a170907724500b00726a909902emr16338729ejc.308.1656708556597; Fri, 01 Jul 2022 13:49:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656708556; cv=none; d=google.com; s=arc-20160816; b=teY7HxP57ZIyMH5KnFIyGaoqm2j3f1U3B+FIaCELuFXRg8lufRg9oKUwzxKnSPWBSr v+1syCZ2IuwspEt4/xczW/GnMgvb6eHHQGv6kSraVJBmXNEiJMa3SOFsYU2Mq3c7EjEZ q6z/BCve5scGiw2nt5kSIb3oubrwTpx5RVodQ3fVGpTtRepg6r8OhszryIDOa0qJLvhS wszcN59d3tPZFkRKb0d27AUaM0DUsxKVKrd0fLJTAfcbaGVAK8bkWEOclcHela/zz/Oc oyulQAJL8DqG3FrrbZE3TP2KmGriyftnWEjv7HECwn7h2akclkcZlm4ih0N0YGv2fLCY 6hvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:references :in-reply-to:message-id:dkim-signature:delivered-to; bh=MFlPt/r9280psJ8ZmzwzVt4dzY0qDeh9DF1LVMupCV0=; b=075OfGuTW9jeovDCW6hyCNiuxaygeWwRUkGFAk7a3SkYrqmr0PsZL3M1nFIoUSTI48 w6N38VvNDNtK5hGsfHUcWtufsEsP1m2N3xhyVL3X6XNg/2FQggHuOxb8Lh6hQJUTHHVJ g/+PY5bpm4WVG77dgxhv2GOQnHYmk4GtgQAAHP8XmXlv8TGt20QSLHGe2it0f4F/XLuC XQSu55EaoSdGFwisqmPECVs242NKKoWS4eV3om1viXrPIDunjfAIG/V4SAIPftHHW4ah DpgkNJ+rT9Azw3M7kmo/VEC6yaC7XaFC76ZGWAOsYnuBoAYFxotOJPCtgEMRy9+yk+xq QDiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=JLzDVP1z; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mp3-20020a1709071b0300b0072a4d2a1a4dsi2312702ejc.767.2022.07.01.13.49.14; Fri, 01 Jul 2022 13:49:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=JLzDVP1z; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 677D268B64E; Fri, 1 Jul 2022 23:49:10 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1E0FB68B64E for ; Fri, 1 Jul 2022 23:48:59 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id n10so3459996plp.0; Fri, 01 Jul 2022 13:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0WRm+MDbonSBjaY4EDTZWtIco31h9G9OEqKPzdK0j7U=; b=JLzDVP1zKJyntFN23+cubpWR6MQlAzpI8FKhUh3/P4Xmdc9QBsn5zlugaClqpM7SSF jxQMj3A0C1qchE8nrHK9n3/EqR5egnr12HxhZ/lclMS2r3V/APxI6uNqptCTAOjQLbnF o6166n7R9s/KOVFBFAAAb01tZqUllrvBQ+ETJVYdr9WiVRjzH4Tm5jw4z2l1ZVnMq9HX 9HS1COQdJ0FXRDvfd72BtfkP3DgYI3g1r7FGpkwM0iiPnUvK/ar0e/uxeOpDE67iDsq/ drI/V4Q584U3U/PrNv6ATdbawmkr+oy2m2KBEW0hddnvS3chhs75U8NJSc/vypy56WSb 7Q5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0WRm+MDbonSBjaY4EDTZWtIco31h9G9OEqKPzdK0j7U=; b=BHhjPWuN4K9qy6rVWpMRxHXZ4txBOmZAKVM3NgMosZETcQPLDWamANp8Usc4r+Rp4h bcRlZciTi0hI3WIYcJhTN8ZuWvUieJ48ZTVLweQwqq2NbBxxgZ6gwVs13qj2eaX9Hj3O +zkqDPqtpqF3Pjmd4jiI+R8EfYUbTY9UXjrdpiKRPQT4+Jk/tddgrY74MdpNHnguZVO0 EKFhZs3+/2FsAsgJ/7179npu0PWb6WgCAuiQ9TNLB3ewo2gsfKoa1yM0PwqgGHnyEdRe JyAc0ksB1v2VlP2NyDew6khqTeuNHkUIp7kJoKalTF8JW/aRzLeIPIEznHFWk1P1zC+D eyjA== X-Gm-Message-State: AJIora+u70xfs5SvX7pB5vHJNQ+rb3iGlwm3Nsv5AVztzs3FdQKyeAP7 iEXwjYSOK/NMUuxb/gAzVAqmCqHuh+dlKg== X-Received: by 2002:a17:90b:1808:b0:1ec:9559:3060 with SMTP id lw8-20020a17090b180800b001ec95593060mr21057888pjb.163.1656708537100; Fri, 01 Jul 2022 13:48:57 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id c4-20020a170903234400b0016b9dbcbd51sm6325227plh.94.2022.07.01.13.48.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Jul 2022 13:48:56 -0700 (PDT) Message-Id: In-Reply-To: References: From: ffmpegagent Date: Fri, 01 Jul 2022 20:48:48 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v5 0/6] Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Kieran Kunhya , softworkz , "Xiang, Haihao" , Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jhQN12oilob/ Missing SEI information has always been a major drawback when using the QSV decoders. I used to think that there's no chance to get at the data without explicit implementation from the MSDK side (or doing something weird like parsing in parallel). It turned out that there's a hardly known api method that provides access to all SEI (h264/hevc) or user data (mpeg2video). This allows to get things like closed captions, frame packing, display orientation, HDR data (mastering display, content light level, etc.) without having to rely on those data being provided by the MSDK as extended buffers. The commit "Implement SEI parsing for QSV decoders" includes some hard-coded workarounds for MSDK bugs which I reported: https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311 But that doesn't help. Those bugs exist and I'm sharing my workarounds, which are empirically determined by testing a range of files. If someone is interested, I can provide private access to a repository where we have been testing this. Alternatively, I could also leave those workarounds out, and just skip those SEI types. In a previous version of this patchset, there was a concern that payload data might need to be re-ordered. Meanwhile I have researched this carefully and the conclusion is that this is not required. My detailed analysis can be found here: https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932 v4 * add new dependencies in makefile Now, build still works when someone uses configure --disable-decoder=h264 --disable-decoder=hevc --disable-decoder=mpegvideo --disable-decoder=mpeg1video --disable-decoder=mpeg2video --enable-libmfx v3 * frame.h: clarify doc text for av_frame_copy_side_data() v2 * qsvdec: make error handling consistent and clear * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants * hevcdec: rename function to ff_hevc_set_side_data(), add doc text v3 * qsvdec: fix c/p error softworkz (6): avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() avcodec/vpp_qsv: Copy side data from input to output frame avcodec/mpeg12dec: make mpeg_decode_user_data() accessible avcodec/hevcdec: make set_side_data() accessible avcodec/h264dec: make h264_export_frame_props() accessible avcodec/qsvdec: Implement SEI parsing for QSV decoders doc/APIchanges | 4 + libavcodec/Makefile | 6 +- libavcodec/h264_slice.c | 98 ++++++++------- libavcodec/h264dec.h | 2 + libavcodec/hevcdec.c | 117 +++++++++--------- libavcodec/hevcdec.h | 9 ++ libavcodec/hevcdsp.c | 4 + libavcodec/mpeg12.h | 28 +++++ libavcodec/mpeg12dec.c | 40 +----- libavcodec/qsvdec.c | 234 +++++++++++++++++++++++++++++++++++ libavfilter/qsvvpp.c | 6 + libavfilter/vf_overlay_qsv.c | 19 ++- libavutil/frame.c | 67 ++++++---- libavutil/frame.h | 32 +++++ libavutil/version.h | 2 +- 15 files changed, 494 insertions(+), 174 deletions(-) base-commit: 6a82412bf33108111eb3f63076fd5a51349ae114 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v5 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-31/softworkz/submit_qsv_sei-v5 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31 Range-diff vs v4: 1: 7656477360 = 1: 7656477360 avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() 2: 06976606c5 = 2: 06976606c5 avcodec/vpp_qsv: Copy side data from input to output frame 3: 320a8a535c = 3: 320a8a535c avcodec/mpeg12dec: make mpeg_decode_user_data() accessible 4: e58ad6564f = 4: e58ad6564f avcodec/hevcdec: make set_side_data() accessible 5: a57bfaebb9 = 5: 4c0b6eb4cb avcodec/h264dec: make h264_export_frame_props() accessible 6: 3f2588563e ! 6: 19bc00be4d avcodec/qsvdec: Implement SEI parsing for QSV decoders @@ Commit message Signed-off-by: softworkz + ## libavcodec/Makefile ## +@@ libavcodec/Makefile: OBJS-$(CONFIG_MSS34DSP) += mss34dsp.o + OBJS-$(CONFIG_PIXBLOCKDSP) += pixblockdsp.o + OBJS-$(CONFIG_QPELDSP) += qpeldsp.o + OBJS-$(CONFIG_QSV) += qsv.o +-OBJS-$(CONFIG_QSVDEC) += qsvdec.o ++OBJS-$(CONFIG_QSVDEC) += qsvdec.o h264_slice.o h264_cabac.o h264_cavlc.o \ ++ h264_direct.o h264_mb.o h264_picture.o h264_loopfilter.o \ ++ h264dec.o h264_refs.o cabac.o hevcdec.o hevc_refs.o \ ++ hevc_filter.o hevc_cabac.o hevc_mvs.o hevcpred.o hevcdsp.o \ ++ h274.o dovi_rpu.o mpeg12dec.o + OBJS-$(CONFIG_QSVENC) += qsvenc.o + OBJS-$(CONFIG_RANGECODER) += rangecoder.o + OBJS-$(CONFIG_RDFT) += rdft.o + + ## libavcodec/hevcdsp.c ## +@@ + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + ++#include "config_components.h" ++ + #include "hevcdsp.h" + + static const int8_t transform[32][32] = { +@@ libavcodec/hevcdsp.c: int i = 0; + break; + } + ++#if CONFIG_HEVC_DECODER + #if ARCH_AARCH64 + ff_hevc_dsp_init_aarch64(hevcdsp, bit_depth); + #elif ARCH_ARM +@@ libavcodec/hevcdsp.c: int i = 0; + #elif ARCH_LOONGARCH + ff_hevc_dsp_init_loongarch(hevcdsp, bit_depth); + #endif ++#endif + } + ## libavcodec/qsvdec.c ## @@ #include "hwconfig.h"