From patchwork Wed Jun 1 18:01:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 36032 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp3581590pzj; Wed, 1 Jun 2022 11:02:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFZDaS+e9JJ6H16lBz2Wz3BJXNJLQYq6SBz32JO/nkKkyw1Rz7P6c+wHM6PsY3xv5aFcD0 X-Received: by 2002:a17:906:60cd:b0:6fd:d95c:3e34 with SMTP id f13-20020a17090660cd00b006fdd95c3e34mr709306ejk.250.1654106577811; Wed, 01 Jun 2022 11:02:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654106577; cv=none; d=google.com; s=arc-20160816; b=gNtioDWjyBc4nCl3GPgOnEyuDGPZCG7QO30dcJuGLXCrSF7FYhBAtwaaWJf44Jwe20 26Q7ND1DEPPr+27aZHYbC7AUCzHl5fTAA3cGvoVY3PrqskkmHy2R5GHUfD2MKLE6gRaR Afc/p835ELQ82nHvYnQqu2az3VG2ZUKOrN7qBQYQy7/uWTEVxjUEavkGE3uOvtaR+/AV On0Jp2GQey8QSKsV6Cb7/QF8sltuHuCtwY5Rm/8fRQ3uriVm8+mze9/G3KFKkag+pMtw LuExToRS28c/Fp9hg5z6F9HBecSJ/vDCs/1hmb4dvwml2KpNmDNi7ntQcTNg+O2N3OKb pdpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=WP7bVJ9oi2yu6IFOCtmhFRM/pXcA6dJccQmx5L8uNVs=; b=ZD2xc+hW4AbqBKR1l9wJXVijzg4Q8K7ZmFidJE+yIK+mzHEokWu0prY1Gke7D+g+QP JMUzujB806gHvqicnRVrukr5B5KW4IeUwLmsqKSiLlgRWT619la1S6QGdXARspzySSxQ XiBRhMQVOgFxmYkqPa3KKcg/Ay+/KEh5WkfIEBvJx3Mp2HS93nXheaIMe9Wk5LCgygzt lKLk2+cjKSXabT1HheyFMV/i+q9HWMx4dlJoLxxuOznIoVwl7UOXc11kuE/K88gbiy4J 6cnIyBoLHJQhEADoFScv/KPQEChT1GABwccCGh79ftWQtuNpzUeXUgB1FzYay5m/UVDt kIcg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=WNzDOtqv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id he20-20020a1709073d9400b006f395c3f118si2519913ejc.591.2022.06.01.11.02.57; Wed, 01 Jun 2022 11:02:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=WNzDOtqv; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6E3C168B7B0; Wed, 1 Jun 2022 21:02:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CE7AE68B766 for ; Wed, 1 Jun 2022 21:01:56 +0300 (EEST) Received: by mail-pf1-f179.google.com with SMTP id w21so2700704pfc.0; Wed, 01 Jun 2022 11:01:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ztkKdZgzNF/YIWvsr5q4Ph/L1tHRIac1vKaNmEnch3I=; b=WNzDOtqv133zNUwFI3aAD39Te63Lpk62ElFCZXQNLtDPE17Ny0tONOkbTSARX6EL5Z MmVX2lDLRtuPuvELVn9k91PpacN1hRTwHaEMx7n1B76Ys0bJItIoozOYmcFUQ2+ZJgt2 DdNo3Tn8IrGJvpoxKAmnfwbvXA/SpURC2yOTFwmxfza1B/MIgB17LQIozjl67J5f4Avg Ow4rkcKB1+fmvdM4JKfV0S5bqegViouTMtVsV+GYatuUAqYqkGhnqMDBNBki4G5kRd5i WFISpXr7lfSsaloeZIEQmR08H55AKF2oJUWi39iIrhcj4GhZuYWmMHFrqIbjBhyqWyIo 3ksg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ztkKdZgzNF/YIWvsr5q4Ph/L1tHRIac1vKaNmEnch3I=; b=XJ2TFDoTV3pGSLsRup6QAyIMxJl5zULL1SGg9P8z4hAgsJw6ItnKVVNy9aocKTJ+qX thQIjIHSj8BXi5DLdW56AjesNnM8HTRDyIdqaKx6ET1MVlXlfYVm9uadjOCG8ml5RsRb rTtzc0/Fo1uA+YxG0CvnLTwuqz3JsodVEmSLdwfztJUAy9b8lWiw4D/ZVqzz6SkQgT8R HWdx8H0kwaHVt2bxu6FN/VNd4aUSpVclnYAAFUYxc4V/dNlpemnpzUDZuP+ozQfoJWSl yzKWKMr7xeQYDYnKo+XeDkGLQZTYNBSbiuLnhWW+PsUD9DDeEE7st2Js87Y79+rbrjKB gfRQ== X-Gm-Message-State: AOAM530+GNBiwl2ElkM0NeLEXXKd2SGqGXilfPemc11M9BwkK7Bz/XPP QweTnZBURd3SZ9O2uxsuQBSevG3GaN4= X-Received: by 2002:a62:1784:0:b0:51b:bc40:28a7 with SMTP id 126-20020a621784000000b0051bbc4028a7mr746836pfx.55.1654106515212; Wed, 01 Jun 2022 11:01:55 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id c22-20020a170902b69600b001639f038250sm1891921pls.220.2022.06.01.11.01.54 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Jun 2022 11:01:54 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <27c3dded4d87cf89176bd460532564ed15ef4d98.1654106506.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Wed, 01 Jun 2022 18:01:46 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v3 6/6] avcodec/qsvdec: Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz , "Xiang, Haihao" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 2ejTjMrjeTbB From: softworkz Signed-off-by: softworkz --- libavcodec/qsvdec.c | 234 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c index 5fc5bed4c8..e854f363ec 100644 --- a/libavcodec/qsvdec.c +++ b/libavcodec/qsvdec.c @@ -49,6 +49,12 @@ #include "hwconfig.h" #include "qsv.h" #include "qsv_internal.h" +#include "h264dec.h" +#include "h264_sei.h" +#include "hevcdec.h" +#include "hevc_ps.h" +#include "hevc_sei.h" +#include "mpeg12.h" static const AVRational mfx_tb = { 1, 90000 }; @@ -60,6 +66,8 @@ static const AVRational mfx_tb = { 1, 90000 }; AV_NOPTS_VALUE : pts_tb.num ? \ av_rescale_q(mfx_pts, mfx_tb, pts_tb) : mfx_pts) +#define PAYLOAD_BUFFER_SIZE 65535 + typedef struct QSVAsyncFrame { mfxSyncPoint *sync; QSVFrame *frame; @@ -101,6 +109,9 @@ typedef struct QSVContext { mfxExtBuffer **ext_buffers; int nb_ext_buffers; + + mfxU8 payload_buffer[PAYLOAD_BUFFER_SIZE]; + Mpeg1Context mpeg_ctx; } QSVContext; static const AVCodecHWConfigInternal *const qsv_hw_configs[] = { @@ -599,6 +610,210 @@ static int qsv_export_film_grain(AVCodecContext *avctx, mfxExtAV1FilmGrainParam return 0; } #endif +static int find_start_offset(mfxU8 data[4]) +{ + if (data[0] == 0 && data[1] == 0 && data[2] == 1) + return 3; + + if (data[0] == 0 && data[1] == 0 && data[2] == 0 && data[3] == 1) + return 4; + + return 0; +} + +static int parse_sei_h264(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + H264SEIContext sei = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + start = find_start_offset(payload.Data); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else { + ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + } + + if (out) + return ff_h264_export_frame_props(avctx, &sei, NULL, out); + + return 0; +} + +static int parse_sei_hevc(AVCodecContext* avctx, QSVContext* q, QSVFrame* out) +{ + HEVCSEI sei = { 0 }; + HEVCParamSets ps = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxFrameSurface1 *surface = &out->surface; + mfxU64 ts; + int ret, has_logged = 0; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + if (!has_logged) { + has_logged = 1; + av_log(avctx, AV_LOG_VERBOSE, "-----------------------------------------\n"); + av_log(avctx, AV_LOG_VERBOSE, "Start reading SEI - payload timestamp: %llu - surface timestamp: %llu\n", ts, surface->Data.TimeStamp); + } + + if (ts != surface->Data.TimeStamp) { + av_log(avctx, AV_LOG_WARNING, "GetPayload timestamp (%llu) does not match surface timestamp: (%llu)\n", ts, surface->Data.TimeStamp); + } + + start = find_start_offset(payload.Data); + + av_log(avctx, AV_LOG_VERBOSE, "parsing SEI type: %3d Numbits %3d Start: %d\n", payload.Type, payload.NumBit, start); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + case SEI_TYPE_MASTERING_DISPLAY_COLOUR_VOLUME: + // There seems to be a bug in MSDK + payload.NumBit -= 8; + + break; + case SEI_TYPE_CONTENT_LIGHT_LEVEL_INFO: + // There seems to be a bug in MSDK + payload.NumBit = 48; + + break; + case SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35: + // There seems to be a bug in MSDK + if (payload.NumBit == 552) + payload.NumBit = 528; + break; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else { + ret = ff_hevc_decode_nal_sei(&gb, avctx, &sei, &ps, HEVC_NAL_SEI_PREFIX); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + } + + if (has_logged) { + av_log(avctx, AV_LOG_VERBOSE, "End reading SEI\n"); + } + + if (out && out->frame) + return ff_hevc_set_side_data(avctx, &sei, NULL, out->frame); + + return 0; +} + +static int parse_sei_mpeg12(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + Mpeg1Context *mpeg_ctx = &q->mpeg_ctx; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + + memset(payload.Data, 0, payload.BufSize); + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + start = find_start_offset(payload.Data); + + start++; + + ff_mpeg_decode_user_data(avctx, mpeg_ctx, &payload.Data[start], (int)((payload.NumBit + 7) / 8) - start); + + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d start %d -> %.s\n", payload.Type, payload.NumBit, start, (char *)(&payload.Data[start])); + } + + if (!out) + return 0; + + if (mpeg_ctx->a53_buf_ref) { + + AVFrameSideData *sd = av_frame_new_side_data_from_buf(out, AV_FRAME_DATA_A53_CC, mpeg_ctx->a53_buf_ref); + if (!sd) + av_buffer_unref(&mpeg_ctx->a53_buf_ref); + mpeg_ctx->a53_buf_ref = NULL; + } + + if (mpeg_ctx->has_stereo3d) { + AVStereo3D *stereo = av_stereo3d_create_side_data(out); + if (!stereo) + return AVERROR(ENOMEM); + + *stereo = mpeg_ctx->stereo3d; + mpeg_ctx->has_stereo3d = 0; + } + + if (mpeg_ctx->has_afd) { + AVFrameSideData *sd = av_frame_new_side_data(out, AV_FRAME_DATA_AFD, 1); + if (!sd) + return AVERROR(ENOMEM); + + *sd->data = mpeg_ctx->afd; + mpeg_ctx->has_afd = 0; + } + + return 0; +} static int qsv_decode(AVCodecContext *avctx, QSVContext *q, AVFrame *frame, int *got_frame, @@ -636,6 +851,8 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, insurf, &outsurf, sync); if (ret == MFX_WRN_DEVICE_BUSY) av_usleep(500); + else if (avctx->codec_id == AV_CODEC_ID_MPEG2VIDEO) + parse_sei_mpeg12(avctx, q, NULL); } while (ret == MFX_WRN_DEVICE_BUSY || ret == MFX_ERR_MORE_SURFACE); @@ -677,6 +894,23 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, return AVERROR_BUG; } + switch (avctx->codec_id) { + case AV_CODEC_ID_MPEG2VIDEO: + ret = parse_sei_mpeg12(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_H264: + ret = parse_sei_h264(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_HEVC: + ret = parse_sei_hevc(avctx, q, out_frame); + break; + default: + ret = 0; + } + + if (ret < 0) + av_log(avctx, AV_LOG_ERROR, "Error parsing SEI data: %d\n", ret); + out_frame->queued += 1; aframe = (QSVAsyncFrame){ sync, out_frame };