From patchwork Wed Jun 1 09:06:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 34739 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp3309412pzj; Wed, 1 Jun 2022 02:06:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwqguNdhLXX3qXOQ1pZxidSMjso2KHrHzFCdmZ9v3uJszcbKnCzVs6TaIKu47eObu9sGmPr X-Received: by 2002:aa7:cad4:0:b0:428:715f:5ce4 with SMTP id l20-20020aa7cad4000000b00428715f5ce4mr70850444edt.124.1654074394111; Wed, 01 Jun 2022 02:06:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654074394; cv=none; d=google.com; s=arc-20160816; b=p6eLVliiwkc8/YzxiS55Qk8H+2CrYwZv4EGHuyj8hJjuHdbZNHW2Q7A4bHf9H/Kd4M oVpt0AapPjN8m3J7qP4r4tYFwPflEfOiw8YQo79N/9uifOtjusZnjyuGe+SdV7jILy2h O8rxZuE0s/X0m0xplFmZfmpzSyUZw6GBzXemB10yl85d6JN1TCpSKcUF8hmiwLFA8OTf nTzGUvFsIBaWxZFtc8QzHcqF8wRNs9pN2YcWswj4abOqCMZ7Ojj85/vmrNhN+tLn+Eu0 lD8QwgUkzxu9DD00PprFVqmVtWHjrdPSGZAvsgB8w1L1FtPAGKsrETxw1ciZHUC5Omiv aLNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:references :in-reply-to:message-id:dkim-signature:delivered-to; bh=PMYU6LhgGwzt+kMXtyxdDemOq/V35UQlVuWBzgbJwGg=; b=QNr3CMImOicBdqmBc9mbJI/gw8TIa+2t37HEHKATXzUh+5Qnuec0HH1LqVl3v39twV iEjczJ6bxGuePjohyVj9167vF13AJwKVrkpMKwH3FBEOwQncJ152Lmfg6KOHSgibNzHR WD0J7t8Ine/OY5mhPd9xNw2zXca2VVd72Uuk9z96IpZ+VeZ70jQ2Qc9Pau1Y6K6Eenid 9kBb11BsI/jKw3wiXOP0kT/SwUUlwpAqvtSDjiiZ7EnK7bswtE2A+nWn+IhIcPZIUYkc pK+ueRI4jAmM54G2Ujk+et66vRcrohc2kHrc990tPUOFgQ+k+5lftqB2LqsklISgTeYr FO1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=LUlupwX+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cs11-20020a170906dc8b00b006ff149c360asi1103230ejc.920.2022.06.01.02.06.33; Wed, 01 Jun 2022 02:06:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=LUlupwX+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4448A68B70C; Wed, 1 Jun 2022 12:06:21 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5DFEC68B588 for ; Wed, 1 Jun 2022 12:06:09 +0300 (EEST) Received: by mail-pf1-f175.google.com with SMTP id y196so1396883pfb.6; Wed, 01 Jun 2022 02:06:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6vcFv2j8uhnzpqpMZq9jiv/SU5JJe27te7WRJaR1DxI=; b=LUlupwX+V/HCKtLMlds5ut69Y4BcZ9cNCa7NQduzCBGxTsF1mBQcZuIYERGtrDmpaN GEVuFalv1O8p+Jc2rrJ8nBsYMoDG4WcfMOerOranbcvM9I+6rrjUrWMmkLh/3RRmzpLn UNRgUNhcy0R9GJqDRknr6M6SjVi6HHoumwVdqsZkWP8vzQyHutuDd/SbpwSHJ5sYLM/i qW+ammi3+p/rhMpdxJ3WDktROIF+IVtvZzkKtyhhJzuyljv7J3bCGkZv7jTC234+VR8r 9+fjnbUT4JEbp2s0PlWrQ3sZ7yZ4PZThfZ944R/aPPU5XOxjuX46IyuthgCibUvQdm1p jfsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6vcFv2j8uhnzpqpMZq9jiv/SU5JJe27te7WRJaR1DxI=; b=5LJI1V/1bhepr0Nd1pY2zD/rIFG6Sus69SRZKJqR4Oac+rTjBHa6cYvxj7XHP4w9bS YDwyDdSpa+3axpbUGRGmpjzHT3gKaUUA29tJMaqrMycrXHvB3SqLMecsra8W86SQxt37 JkpTQfEfdmcaHU8/gughUSWBNPqT0ZlitEEd2qE4neQ6nBxjFBRE2riTQehBjpHkIiJ7 xBpiwuiXSds1OUgC9MAcmZ9/z/vLVUxOGjyb33oyRx7dtLMtxgol8qA4IlohAMpgZOqN IOTIjIo0kBGrHAU+ADSUa2Da13gZHAHmgHCJnt0bMinWeghV9g2vpJDUtmR/ron9RQNr Xgaw== X-Gm-Message-State: AOAM533V7p07TY2BKy+QTQb8dsPU+XSWHCWl/BTW2D5H3JGmwID9aH28 eNsRARn/r8BH27+dCniddhNCxpauLEM= X-Received: by 2002:a63:84c6:0:b0:3fb:ea7f:5078 with SMTP id k189-20020a6384c6000000b003fbea7f5078mr15244411pgd.510.1654074367537; Wed, 01 Jun 2022 02:06:07 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id y19-20020a1709029b9300b0015e8d4eb2b4sm974447plp.254.2022.06.01.02.06.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Jun 2022 02:06:07 -0700 (PDT) Message-Id: In-Reply-To: References: From: ffmpegagent Date: Wed, 01 Jun 2022 09:06:00 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v2 0/6] Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz , "Xiang, Haihao" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: dwfc3SyTDsnx Missing SEI information has always been a major drawback when using the QSV decoders. I used to think that there's no chance to get at the data without explicit implementation from the MSDK side (or doing something weird like parsing in parallel). It turned out that there's a hardly known api method that provides access to all SEI (h264/hevc) or user data (mpeg2video). This allows to get things like closed captions, frame packing, display orientation, HDR data (mastering display, content light level, etc.) without having to rely on those data being provided by the MSDK as extended buffers. The commit "Implement SEI parsing for QSV decoders" includes some hard-coded workarounds for MSDK bugs which I reported: https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311 But that doesn't help. Those bugs exist and I'm sharing my workarounds, which are empirically determined by testing a range of files. If someone is interested, I can provide private access to a repository where we have been testing this. Alternatively, I could also leave those workarounds out, and just skip those SEI types. In a previous version of this patchset, there was a concern that payload data might need to be re-ordered. Meanwhile I have researched this carefully and the conclusion is that this is not required. My detailed analysis can be found here: https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932 v2 * qsvdec: make error handling consistent and clear * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants * hevcdec: rename function to ff_hevc_set_side_data(), add doc text softworkz (6): avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() avcodec/vpp_qsv: Copy side data from input to output frame avcodec/mpeg12dec: make mpeg_decode_user_data() accessible avcodec/hevcdec: make set_side_data() accessible avcodec/h264dec: make h264_export_frame_props() accessible avcodec/qsvdec: Implement SEI parsing for QSV decoders doc/APIchanges | 4 + libavcodec/h264_slice.c | 98 ++++++++------- libavcodec/h264dec.h | 2 + libavcodec/hevcdec.c | 117 +++++++++--------- libavcodec/hevcdec.h | 9 ++ libavcodec/mpeg12.h | 28 +++++ libavcodec/mpeg12dec.c | 40 +----- libavcodec/qsvdec.c | 234 +++++++++++++++++++++++++++++++++++ libavfilter/qsvvpp.c | 6 + libavfilter/vf_overlay_qsv.c | 19 ++- libavutil/frame.c | 67 ++++++---- libavutil/frame.h | 32 +++++ libavutil/version.h | 2 +- 13 files changed, 485 insertions(+), 173 deletions(-) base-commit: b033913d1c5998a29dfd13e9906dd707ff6eff12 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v2 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-31/softworkz/submit_qsv_sei-v2 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31 Range-diff vs v1: 1: 4ee6cb47db = 1: 4ee6cb47db avutil/frame: Add av_frame_copy_side_data() and av_frame_remove_all_side_data() 2: 3152156c97 = 2: 3152156c97 avcodec/vpp_qsv: Copy side data from input to output frame 3: 8082c3ab84 = 3: 8082c3ab84 avcodec/mpeg12dec: make mpeg_decode_user_data() accessible 4: 485d7f913d ! 4: 306bdaa39c avcodec/hevcdec: make set_side_data() accessible @@ libavcodec/hevcdec.c: error: } -static int set_side_data(HEVCContext *s) -+int ff_set_side_data(AVCodecContext *logctx, HEVCSEI *sei, HEVCContext *s, AVFrame *out) ++int ff_hevc_set_side_data(AVCodecContext *logctx, HEVCSEI *sei, HEVCContext *s, AVFrame *out) { - AVFrame *out = s->ref->frame; - int ret; @@ libavcodec/hevcdec.c: static int hevc_frame_start(HEVCContext *s) } - ret = set_side_data(s); -+ ret = ff_set_side_data(s->avctx, &s->sei, s, s->ref->frame); ++ ret = ff_hevc_set_side_data(s->avctx, &s->sei, s, s->ref->frame); if (ret < 0) goto fail; @@ libavcodec/hevcdec.h: void ff_hevc_hls_residual_coding(HEVCContext *s, int x0, i void ff_hevc_hls_mvd_coding(HEVCContext *s, int x0, int y0, int log2_cb_size); -+int ff_set_side_data(AVCodecContext *logctx, HEVCSEI *sei, HEVCContext *s, AVFrame *out); ++/** ++ * Set the decodec side data to an AVFrame. ++ * @logctx context for logging. ++ * @sei HEVCSEI decoding context, must not be NULL. ++ * @s HEVCContext, can be NULL. ++ * @return < 0 on error, 0 otherwise. ++ */ ++int ff_hevc_set_side_data(AVCodecContext *logctx, HEVCSEI *sei, HEVCContext *s, AVFrame *out); + extern const uint8_t ff_hevc_qpel_extra_before[4]; extern const uint8_t ff_hevc_qpel_extra_after[4]; 5: fb5c3df8e5 = 5: 16f5dfbfd1 avcodec/h264dec: make h264_export_frame_props() accessible 6: dcf08cd7b7 ! 6: 23de6d2774 avcodec/qsvdec: Implement SEI parsing for QSV decoders @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); -+ if (ret != MFX_ERR_NONE) { -+ av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); -+ return ret; ++ if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { ++ av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); ++ return 0; + } ++ if (ret != MFX_ERR_NONE) ++ return ret; + -+ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { ++ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; -+ } + + start = find_start_offset(payload.Data); + @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) -+ av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader"); -+ else ++ av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); ++ else { + ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); + -+ if (ret < 0) -+ av_log(avctx, AV_LOG_WARNING, "Error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); -+ else -+ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); ++ if (ret < 0) ++ av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); ++ else ++ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); ++ } + } + + if (out) @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); -+ if (ret != MFX_ERR_NONE) { -+ av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); ++ if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { ++ av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } ++ if (ret != MFX_ERR_NONE) ++ return ret; + -+ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { ++ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; -+ } + + if (!has_logged) { + has_logged = 1; @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) -+ av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader"); -+ else -+ ret = ff_hevc_decode_nal_sei(&gb, avctx, &sei, &ps, HEVC_NAL_SEI_PREFIX); ++ av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); ++ else { ++ ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); + -+ if (ret < 0) -+ av_log(avctx, AV_LOG_WARNING, "error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); -+ else -+ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); ++ if (ret < 0) ++ av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); ++ else ++ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); ++ } + } + + if (has_logged) { @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + } + + if (out && out->frame) -+ return ff_set_side_data(avctx, &sei, NULL, out->frame); ++ return ff_hevc_set_side_data(avctx, &sei, NULL, out->frame); + + return 0; +} @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + + memset(payload.Data, 0, payload.BufSize); + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); -+ if (ret != MFX_ERR_NONE) { -+ av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); -+ return ret; ++ if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { ++ av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); ++ return 0; + } ++ if (ret != MFX_ERR_NONE) ++ return ret; + -+ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { ++ if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; -+ } + + start = find_start_offset(payload.Data); + @@ libavcodec/qsvdec.c: static int qsv_export_film_grain(AVCodecContext *avctx, mfx + + ff_mpeg_decode_user_data(avctx, mpeg_ctx, &payload.Data[start], (int)((payload.NumBit + 7) / 8) - start); + -+ if (ret < 0) -+ av_log(avctx, AV_LOG_WARNING, "error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); -+ else -+ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d start %d -> %.s\n", payload.Type, payload.NumBit, start, (char *)(&payload.Data[start])); ++ av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d start %d -> %.s\n", payload.Type, payload.NumBit, start, (char *)(&payload.Data[start])); + } + + if (!out) @@ libavcodec/qsvdec.c: static int qsv_decode(AVCodecContext *avctx, QSVContext *q, insurf, &outsurf, sync); if (ret == MFX_WRN_DEVICE_BUSY) av_usleep(500); -+ else if (avctx->codec_id == AV_CODEC_ID_MPEG1VIDEO || avctx->codec_id == AV_CODEC_ID_MPEG2VIDEO) ++ else if (avctx->codec_id == AV_CODEC_ID_MPEG2VIDEO) + parse_sei_mpeg12(avctx, q, NULL); } while (ret == MFX_WRN_DEVICE_BUSY || ret == MFX_ERR_MORE_SURFACE); @@ libavcodec/qsvdec.c: static int qsv_decode(AVCodecContext *avctx, QSVContext *q, } + switch (avctx->codec_id) { -+ case AV_CODEC_ID_MPEG1VIDEO: + case AV_CODEC_ID_MPEG2VIDEO: + ret = parse_sei_mpeg12(avctx, q, out_frame->frame); + break; @@ libavcodec/qsvdec.c: static int qsv_decode(AVCodecContext *avctx, QSVContext *q, + } + + if (ret < 0) -+ av_log(avctx, AV_LOG_ERROR, "Error parsing SEI data\n"); ++ av_log(avctx, AV_LOG_ERROR, "Error parsing SEI data: %d\n", ret); + out_frame->queued += 1;