From patchwork Sun Jun 26 23:41:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 34758 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp1695965pzh; Sun, 26 Jun 2022 16:42:12 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tO3PLnMp97VC962l6cj/aOw8nO20AGVCKlqdGbUDHqvOagQguwKAlMTOWNPCf79KngxP3R X-Received: by 2002:a05:6402:27c6:b0:435:d24a:d061 with SMTP id c6-20020a05640227c600b00435d24ad061mr13445772ede.427.1656286932765; Sun, 26 Jun 2022 16:42:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656286932; cv=none; d=google.com; s=arc-20160816; b=fR34mmHtePR6bQTxYH5cawavn7v4X9FRyreie8+L00cynEtR3foBznac2HavdxbvFJ Oo0teg6NoRzXbqxnqPqAJXhHje5aOuNthGPx39J3o1DAKyjEYiVeq1VBVnBF1MoBLurn 9w/zc6SpWDhSXbKtevEDBLCctNLTwidmd+xGKmvWJ+tTK/iypAe4aKQw7cf9EhJk2zbF ALym9wV+bI4+Uu0/CiQD1pCrqUd0Oxo66UAcDvQfWxr/tIHaGz4r91u8T+AOXxkz8sVv 89+wNozleTtvWagcV/kF/hrFbacrbWFTr8rdSQADLDPMtyJzsRHctJufA6uL75HbTqtJ IbXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:references :in-reply-to:message-id:dkim-signature:delivered-to; bh=0/NcTyQDOz/n+RiEcVrUEtxnaR1CW+KtEdtpe/6RpYg=; b=NdK9yYUDV7zEXXYtWoWFahh9Fjo5wW+vO5eRk4D7jTvIsl+inKbmhMrPLMs0avpjzg yvennSmBM6b8Y/+UmCodSl18l2B4XnWoQLh6m9rwSYVw1bNPuU5zaN5oi1TXUApFmLrm wGuNoXNaJD4hDu9EmwVwXzBRe6hx31xdnf8L2/EXfbI0HDJmHBH7rWqoGmsrUkoF3PcL dREtJS5AoH560TAyCYF/VgrLtriU4oeZaMCG5Wwtj4lm1hlxSuEKdDZWMJdKu/jp7W3c 204Z4VDQXn225uMKu7gOL9b9reiJlfdcPTNL6ZVNesCssxF+LRa30rslgXmNAN9BoYHA v3pA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=DtdNKeKU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q23-20020aa7d457000000b0043582787ec4si9704237edr.85.2022.06.26.16.42.12; Sun, 26 Jun 2022 16:42:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=DtdNKeKU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6A25C68B881; Mon, 27 Jun 2022 02:41:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 663FC68B41C for ; Mon, 27 Jun 2022 02:41:32 +0300 (EEST) Received: by mail-pj1-f43.google.com with SMTP id b12-20020a17090a6acc00b001ec2b181c98so10676482pjm.4; Sun, 26 Jun 2022 16:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=t62x7cV1zvVNr2mnFGKkOwvvTMihNVFtOBEB4J6vHqQ=; b=DtdNKeKU2tqDodpZJiD+A7gU2wD2Z8emIpiQrcvxJFDrJWcBOxcJo/UFlUIVB+dA7c X7WfDOy3pYghiXmZZCiZg0jGYfXgakgxw37PDsaORfeOb5N2StH5LB9NHSeKbQUZm/zX U1txpP2nBLk0QdEa50714FTf20Sfh/SoxBv0buWjUZXiLDnRs7Wmi+yiRqPKN79T3COG rE0KmNtkn14uk3mZFM350QC3SCHioEWPfwaYvjAaGTP6gMqQO5vl0Qd53X4itG00wHGP +Mmvu7SMYM8ftE9SyhayaU10AAKNLdEemKCFC1NLat2gOsvRdwhN+GX5IXG7Pk8FgZWD +tsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=t62x7cV1zvVNr2mnFGKkOwvvTMihNVFtOBEB4J6vHqQ=; b=Fk04MHkTaSMmNSHY0LA7YEUDPUkpg23FXK+cWeYVk7uhkxvfNtneMiwSrNqGsD+awL L0W38dag0Qnftxkss3QCMRKzgHf4G+KAUEORTYXX3pQ/5ispyIXI38lwP4HW3ZPcS89T dwtESYYFVplS+ZM+z5vsM1Keiyto+KiDyNV01QegMPnsJidicex4oS4KRxA5udH8vxie iZsgNWBNE2X6dHOuowVTUr+Rb2B+QT3WQ5GM61My5jDXzrDjPIwox/PJ8Ootg3tSg1qH 2YnCpGglE14Y0uix+vfDRCU0+sef+q2L/hpgIS79QoL5SDO1alqrIiZ70LG6DEsX4F5Z EEaw== X-Gm-Message-State: AJIora9rmLpXM8L2dfA64GJsYSjJ9ugHiohg7251u5liBOSKnG6p/3hP my8d69VJ9i/t6TjuQ6Jy5XAIHBgpjIemnA== X-Received: by 2002:a17:902:cecc:b0:16a:416c:3d14 with SMTP id d12-20020a170902cecc00b0016a416c3d14mr11426254plg.73.1656286889771; Sun, 26 Jun 2022 16:41:29 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id u2-20020a1709026e0200b0016a106cb221sm5675378plk.243.2022.06.26.16.41.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 Jun 2022 16:41:29 -0700 (PDT) Message-Id: In-Reply-To: References: From: ffmpegagent Date: Sun, 26 Jun 2022 23:41:22 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 0/6] Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Kieran Kunhya , softworkz , "Xiang, Haihao" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: LJ6LHStgVbdv Missing SEI information has always been a major drawback when using the QSV decoders. I used to think that there's no chance to get at the data without explicit implementation from the MSDK side (or doing something weird like parsing in parallel). It turned out that there's a hardly known api method that provides access to all SEI (h264/hevc) or user data (mpeg2video). This allows to get things like closed captions, frame packing, display orientation, HDR data (mastering display, content light level, etc.) without having to rely on those data being provided by the MSDK as extended buffers. The commit "Implement SEI parsing for QSV decoders" includes some hard-coded workarounds for MSDK bugs which I reported: https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311 But that doesn't help. Those bugs exist and I'm sharing my workarounds, which are empirically determined by testing a range of files. If someone is interested, I can provide private access to a repository where we have been testing this. Alternatively, I could also leave those workarounds out, and just skip those SEI types. In a previous version of this patchset, there was a concern that payload data might need to be re-ordered. Meanwhile I have researched this carefully and the conclusion is that this is not required. My detailed analysis can be found here: https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932 v3 * frame.h: clarify doc text for av_frame_copy_side_data() v2 * qsvdec: make error handling consistent and clear * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants * hevcdec: rename function to ff_hevc_set_side_data(), add doc text v3 * qsvdec: fix c/p error softworkz (6): avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() avcodec/vpp_qsv: Copy side data from input to output frame avcodec/mpeg12dec: make mpeg_decode_user_data() accessible avcodec/hevcdec: make set_side_data() accessible avcodec/h264dec: make h264_export_frame_props() accessible avcodec/qsvdec: Implement SEI parsing for QSV decoders doc/APIchanges | 4 + libavcodec/h264_slice.c | 98 ++++++++------- libavcodec/h264dec.h | 2 + libavcodec/hevcdec.c | 117 +++++++++--------- libavcodec/hevcdec.h | 9 ++ libavcodec/mpeg12.h | 28 +++++ libavcodec/mpeg12dec.c | 40 +----- libavcodec/qsvdec.c | 234 +++++++++++++++++++++++++++++++++++ libavfilter/qsvvpp.c | 6 + libavfilter/vf_overlay_qsv.c | 19 ++- libavutil/frame.c | 67 ++++++---- libavutil/frame.h | 32 +++++ libavutil/version.h | 2 +- 13 files changed, 485 insertions(+), 173 deletions(-) base-commit: 6a82412bf33108111eb3f63076fd5a51349ae114 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v4 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-31/softworkz/submit_qsv_sei-v4 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31 Range-diff vs v3: 1: c442597a35 ! 1: 7656477360 avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() @@ doc/APIchanges: libavutil: 2021-04-27 API changes, most recent first: -+2022-05-26 - xxxxxxxxx - lavu 57.26.100 - frame.h ++2022-05-26 - xxxxxxxxx - lavu 57.28.100 - frame.h + Add av_frame_remove_all_side_data(), av_frame_copy_side_data(), + AV_FRAME_TRANSFER_SD_COPY, and AV_FRAME_TRANSFER_SD_FILTER. + - 2022-05-23 - xxxxxxxxx - lavu 57.25.100 - avutil.h - Deprecate av_fopen_utf8() without replacement. - + 2022-06-12 - xxxxxxxxxx - lavf 59.25.100 - avio.h + Add avio_vprintf(), similar to avio_printf() but allow to use it + from within a function taking a variable argument list as input. ## libavutil/frame.c ## @@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS @@ libavutil/frame.h: int av_frame_copy(AVFrame *dst, const AVFrame *src); + * @param src a frame from which to copy the side data. + * @param flags a combination of AV_FRAME_TRANSFER_SD_* + * -+ * @return >= 0 on success, a negative AVERROR on error. ++ * @return 0 on success, a negative AVERROR on error. + * + * @note This function will create new references to side data buffers in src, + * unless the AV_FRAME_TRANSFER_SD_COPY flag is passed. @@ libavutil/version.h */ #define LIBAVUTIL_VERSION_MAJOR 57 --#define LIBAVUTIL_VERSION_MINOR 25 -+#define LIBAVUTIL_VERSION_MINOR 26 +-#define LIBAVUTIL_VERSION_MINOR 27 ++#define LIBAVUTIL_VERSION_MINOR 28 #define LIBAVUTIL_VERSION_MICRO 100 #define LIBAVUTIL_VERSION_INT AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR, \ 2: 6f50d0bd57 = 2: 06976606c5 avcodec/vpp_qsv: Copy side data from input to output frame 3: f682b1d695 = 3: 320a8a535c avcodec/mpeg12dec: make mpeg_decode_user_data() accessible 4: 995d835233 = 4: e58ad6564f avcodec/hevcdec: make set_side_data() accessible 5: ac8dc06395 = 5: a57bfaebb9 avcodec/h264dec: make h264_export_frame_props() accessible 6: 27c3dded4d = 6: 3f2588563e avcodec/qsvdec: Implement SEI parsing for QSV decoders