From patchwork Wed Jun 1 18:01:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 34740 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp3581088pzj; Wed, 1 Jun 2022 11:02:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJweFIL0mtShQiFLnN11bXVf044J6b3BBm/C7j0v2NN3iUS6q/JcR+VR8AfCaEUhhHeVdhEt X-Received: by 2002:a17:906:6a28:b0:6ff:d8b:955e with SMTP id qw40-20020a1709066a2800b006ff0d8b955emr655668ejc.505.1654106526986; Wed, 01 Jun 2022 11:02:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654106526; cv=none; d=google.com; s=arc-20160816; b=No45oJ9i7I0oVnPK0QQT0VK9baPrSgKZUZt9+10mQArvF4lSaYXcYwzEzrPKsuu6bi q1GAogIXs3xEkK6qGL0oDwK2CdzTCWiuvL/2pQQZKXbu0eu5UnnrAiGCAyCEC3byFxXI nYrHtuL+ZUne1Ohkv+wNrvbmsLGZTQ9mb4FBvqMxb8gCQT4MlT9vogvVCBtnikN2b0Ab e5gSNy81TXvoSJB/dB4z9I9FxtxN6DAVy4VFHJc0mNx/ZEsnzZWKhIkd6FrFIMf4e3el yfHwtpvsDubnmYzEu87nW6p3cy7dABLr3ncx93cD6N3weeUiDNSEFO89AfGRBGBN3ZNk 1hCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:references :in-reply-to:message-id:dkim-signature:delivered-to; bh=Vr1jwOAVK3j+hw5TmIVVQCLHOynhgsOtBA8LIXYRPns=; b=YzJdb82tEBIKsRlPtJVcZlC/blbXhMZ35J2TdWR8McKgSBONlyBj75IlcuMvRjpvCg 9H3Toko+7N2hOi9FqQ61EWxJiRfMCwaZpjfAP/kveQCieHczjtkzPUH90+SeHjt1v4sS /YsOgKxHRSbjOIxcwCQeT2+4jtNjMFklY815Dw0tvRuTsTkNy9FWTwGODhi+G7jOLYib 0NkYtL5ezLPxeOxSn/qCRrnAaqtolhtVdha1zxX46lqmYYlU58Rr8/LKGzT1sZoTPMvN 1ePj0Z6OxhlvYfon4m+gkfnjSKGk8DGCAEbf/vhpuVcVc6WOBS3K00yZztBY7WBAfYcr jpsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=qLQEGowy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mp2-20020a1709071b0200b006ff46987debsi2554788ejc.514.2022.06.01.11.02.04; Wed, 01 Jun 2022 11:02:06 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=qLQEGowy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AA06768B760; Wed, 1 Jun 2022 21:02:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6812068B760 for ; Wed, 1 Jun 2022 21:01:49 +0300 (EEST) Received: by mail-pf1-f171.google.com with SMTP id p8so2653554pfh.8; Wed, 01 Jun 2022 11:01:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZQAzUhupUxgyVknZxAYEH6kikmggcQtePGFHHDRVcKg=; b=qLQEGowy6uVG3NE2KlcafVo9/P8SW6cciYb2BQstlXMLju7LqO48S2WA8esGOccHtJ BjXP9MwEYGLbQXV66+5Msk0yDFgxOxp4vnDvXPvqoLgT4vm8m5lC6zDVSKSwUpgdOsp7 LXWqdi82zvPsd2KMU3KSs4L/GAzKKeg4JSRCtBb2r7JkQBu26mUgajfyVLfl1W79hhxx Run8mDKd1g6ric2G6CZXrY50nu8HLJKjGJ/ND9KSU8hegpP6hwT3AUIPK1k9euIhwN/n X1zVAvAuWvJKD8TsOBzmlX3a00b96CO415jMDbTJ6aDvviOa2nwvRx7w6PG8JqDV72o2 Ci8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZQAzUhupUxgyVknZxAYEH6kikmggcQtePGFHHDRVcKg=; b=n9nihZHbrNvK7RyzSwtmi7wEtQo7rbs23iaApboYfb15NHfoEwO3d49KcejxYmxAKr DbCWhLxznY5fD8Osym3k6JSMqUiiJ32vgVk6uqZIkQWpIotC2b5GKc8bfWA90gHIY3vm twAS8y/1863amMu5s1AdxffNjOU9Yb8WI4C5S4kuvUWhyr9CTUkOtMVNYOS11gvmvgAs /nqLF1Apk3Ori/Nd64UHlfCLwxC2OpQIsu1DU6ETzAokb+Grs7inn/zHb6Pku6hUJO8C wS4/D0X1DZYyao2n7y2rrPNc7VKJ9lGyW/QtxU0uSgW6hNg34eNPqxWU6e3ez3pgFzyb N7cg== X-Gm-Message-State: AOAM532m7QuxUmr5A/3//5j2srgnfRa8sBO3FaympF3LB4vlFeWrZdxr w+f/5rojDTGQS2dad1Mook6Uiv5Z8bw= X-Received: by 2002:a63:f645:0:b0:3fc:9ac3:e6ab with SMTP id u5-20020a63f645000000b003fc9ac3e6abmr489069pgj.309.1654106507727; Wed, 01 Jun 2022 11:01:47 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id c143-20020a621c95000000b005187c4d53c9sm1740973pfc.160.2022.06.01.11.01.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Jun 2022 11:01:47 -0700 (PDT) Message-Id: In-Reply-To: References: From: ffmpegagent Date: Wed, 01 Jun 2022 18:01:40 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 0/6] Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz , "Xiang, Haihao" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UTfg33MZZulu Missing SEI information has always been a major drawback when using the QSV decoders. I used to think that there's no chance to get at the data without explicit implementation from the MSDK side (or doing something weird like parsing in parallel). It turned out that there's a hardly known api method that provides access to all SEI (h264/hevc) or user data (mpeg2video). This allows to get things like closed captions, frame packing, display orientation, HDR data (mastering display, content light level, etc.) without having to rely on those data being provided by the MSDK as extended buffers. The commit "Implement SEI parsing for QSV decoders" includes some hard-coded workarounds for MSDK bugs which I reported: https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311 But that doesn't help. Those bugs exist and I'm sharing my workarounds, which are empirically determined by testing a range of files. If someone is interested, I can provide private access to a repository where we have been testing this. Alternatively, I could also leave those workarounds out, and just skip those SEI types. In a previous version of this patchset, there was a concern that payload data might need to be re-ordered. Meanwhile I have researched this carefully and the conclusion is that this is not required. My detailed analysis can be found here: https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932 v2 * qsvdec: make error handling consistent and clear * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants * hevcdec: rename function to ff_hevc_set_side_data(), add doc text v3 * qsvdec: fix c/p error softworkz (6): avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() avcodec/vpp_qsv: Copy side data from input to output frame avcodec/mpeg12dec: make mpeg_decode_user_data() accessible avcodec/hevcdec: make set_side_data() accessible avcodec/h264dec: make h264_export_frame_props() accessible avcodec/qsvdec: Implement SEI parsing for QSV decoders doc/APIchanges | 4 + libavcodec/h264_slice.c | 98 ++++++++------- libavcodec/h264dec.h | 2 + libavcodec/hevcdec.c | 117 +++++++++--------- libavcodec/hevcdec.h | 9 ++ libavcodec/mpeg12.h | 28 +++++ libavcodec/mpeg12dec.c | 40 +----- libavcodec/qsvdec.c | 234 +++++++++++++++++++++++++++++++++++ libavfilter/qsvvpp.c | 6 + libavfilter/vf_overlay_qsv.c | 19 ++- libavutil/frame.c | 67 ++++++---- libavutil/frame.h | 32 +++++ libavutil/version.h | 2 +- 13 files changed, 485 insertions(+), 173 deletions(-) base-commit: 77b529fbd228fe30a870e3157f051885b436ad92 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v3 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-31/softworkz/submit_qsv_sei-v3 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31 Range-diff vs v2: 1: 4ee6cb47db = 1: c442597a35 avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() 2: 3152156c97 = 2: 6f50d0bd57 avcodec/vpp_qsv: Copy side data from input to output frame 3: 8082c3ab84 = 3: f682b1d695 avcodec/mpeg12dec: make mpeg_decode_user_data() accessible 4: 306bdaa39c = 4: 995d835233 avcodec/hevcdec: make set_side_data() accessible 5: 16f5dfbfd1 = 5: ac8dc06395 avcodec/h264dec: make h264_export_frame_props() accessible 6: 23de6d2774 ! 6: 27c3dded4d avcodec/qsvdec: Implement SEI parsing for QSV decoders @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else { -+ ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); ++ ret = ff_hevc_decode_nal_sei(&gb, avctx, &sei, &ps, HEVC_NAL_SEI_PREFIX); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret);