From patchwork Sun Jun 26 23:41:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 36477 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:8b27:b0:88:1bbf:7fd2 with SMTP id l39csp1695893pzh; Sun, 26 Jun 2022 16:42:03 -0700 (PDT) X-Google-Smtp-Source: AGRyM1vU5EXGZSuhlf2hpD9zvAdTgKT7x9bTGq1E4fugAgenL5JdM/O20hYwVRxi8u7Ze9gDNP3Z X-Received: by 2002:a17:907:6e17:b0:726:2b3c:d373 with SMTP id sd23-20020a1709076e1700b007262b3cd373mr9964739ejc.357.1656286923699; Sun, 26 Jun 2022 16:42:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656286923; cv=none; d=google.com; s=arc-20160816; b=yCBUSOqFgi/U8Bk5GY65labESqhGDZP786HbwH+vdwOhKS6HuCeBjGtEbA13++Hg5Z uXhyNnssE0N/F485W+uqn2oF2ZAooJgtI0wmKiTg9U5WC55CHF4eO7tOy5UGGVKilpJc v/VSFGPCgvxk0qPhmaa5NgOzJ++7XsPtkhGfh7yU0CalMI8AuC6uuRlkD0cKHAD9n5uV JO/VbH2/Q4EcPmhp46u9sJEkzWDA/Zv1i/5kyP0PqBAhmB1Sfri35jyqXEGK8Aq0Cbuk XbaaexgM+H9eLRPtFiHKcf/rNaTQXgWxFHPAt+WWeh2ljMZ+VRXyDHpOT4oDlRg0ioLI l1uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=WP7bVJ9oi2yu6IFOCtmhFRM/pXcA6dJccQmx5L8uNVs=; b=V6qShjGfRfuKd2yJ2Gzg4pRKAQkoRpSMDIcDSjN5djGuPuWbWOYGQZCwJYMR0w0kUF 7+4MDFdv/gZQLQP616Z3IwuGYNt3bUXuc2bZAwDqhEvIafVkEDTViSfGnCLGmtNn5TIY 64jiBzETkWvFzZfeKT1rzcHmkCyQ9GSPyPs+MO2qrP2mCUac3VHAJDJeEhK76Ox2NLfi WDLnINftvArNND6FVynXRaZJKUPQmKzN4SVBla3ZBYGuqVC9F2rIqibJEr6K/eBJau21 Ey1S6lxkmF+1yOGzgq99/rjm09jHJ3rcjBPGhiIQJiBlAK7g66iF/KRUxz/ImTR6xk8f 0nQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=N8JNYNuX; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y11-20020aa7ce8b000000b00435817432fesi10188793edv.308.2022.06.26.16.42.03; Sun, 26 Jun 2022 16:42:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=N8JNYNuX; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6003968B875; Mon, 27 Jun 2022 02:41:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 117FF68B852 for ; Mon, 27 Jun 2022 02:41:37 +0300 (EEST) Received: by mail-pj1-f46.google.com with SMTP id g16-20020a17090a7d1000b001ea9f820449so10671877pjl.5; Sun, 26 Jun 2022 16:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ztkKdZgzNF/YIWvsr5q4Ph/L1tHRIac1vKaNmEnch3I=; b=N8JNYNuXHHXRRTkDDFTpTddl5S7cyJ1LY3HBw1bOLNk7gKWzmamj0tj4+Annyk8HTh lpwHv8iH7Y8qWy2JezLXJbaHng0FCiH2bxfPJ50E+v4McFaCNeJ0+q7o34bM9NtsHC7l P6jfuFG5rPrzEdL5T0mrBVzTbP67jaZSwjTfo17VIM5RcKVYx6zuXm5GKKcngK+vJxEC knW9sXaiDAO/edzWTx9JZxd2VOfhrzE/Vb8QsLHiI+SaCfzhiXQHfX2KcbjPCp9m689R WSDcvx/CQoAAyUOiMLV7TeTvee5FMHsnU+axtFMhu/Mxa5c9COHALrH48O4C54Rm7pd7 90gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ztkKdZgzNF/YIWvsr5q4Ph/L1tHRIac1vKaNmEnch3I=; b=ExvHPQIThYXP4FzwNqa2FqREQBG7z0k+fUePZ9cLbHVUvCFAVKf/tkavqBzTlTowu+ hi15Lk4fIXQJBFIf4ngm6yH/gkXZdhYD8hq5ihd406x26dlbLz4y6wSHPiSXouJRkAZO DgzIO3k81ZBiAWpKzvezMTO+flt5CeNrB8eyRQnNUIzcxF3pDDORRTqt8IcmO844tgIZ r34EDsfzbqqQdQfNQAb4nL6pgcLG+0koVN2+N5P8YuzaXprvjn1wNZAZycR40E4U76P5 FMKWuBwi8pbtAxazv+OgwKAK70zLB+t2bjEmBVLEVqIHva2QvYmM++jWtJkegfi1Vg5V hijw== X-Gm-Message-State: AJIora+uC1gSP9TkxDNPnYndSQHt+muTKGTjQB0rqBRnhGaAdIw9o/Gu aQSprhYL3qNInmWLr3UFuIBS/78sZkonhw== X-Received: by 2002:a17:90b:1c05:b0:1ec:da52:1cdb with SMTP id oc5-20020a17090b1c0500b001ecda521cdbmr12148431pjb.88.1656286895414; Sun, 26 Jun 2022 16:41:35 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id i29-20020a63585d000000b00407e25d4527sm5608363pgm.22.2022.06.26.16.41.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 Jun 2022 16:41:34 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: <3f2588563e96ce105fddde849e86acf7cb3e2dbf.1656286888.git.ffmpegagent@gmail.com> In-Reply-To: References: Date: Sun, 26 Jun 2022 23:41:28 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v4 6/6] avcodec/qsvdec: Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Kieran Kunhya , softworkz , "Xiang, Haihao" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: rVXUVzHllgbb From: softworkz Signed-off-by: softworkz --- libavcodec/qsvdec.c | 234 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c index 5fc5bed4c8..e854f363ec 100644 --- a/libavcodec/qsvdec.c +++ b/libavcodec/qsvdec.c @@ -49,6 +49,12 @@ #include "hwconfig.h" #include "qsv.h" #include "qsv_internal.h" +#include "h264dec.h" +#include "h264_sei.h" +#include "hevcdec.h" +#include "hevc_ps.h" +#include "hevc_sei.h" +#include "mpeg12.h" static const AVRational mfx_tb = { 1, 90000 }; @@ -60,6 +66,8 @@ static const AVRational mfx_tb = { 1, 90000 }; AV_NOPTS_VALUE : pts_tb.num ? \ av_rescale_q(mfx_pts, mfx_tb, pts_tb) : mfx_pts) +#define PAYLOAD_BUFFER_SIZE 65535 + typedef struct QSVAsyncFrame { mfxSyncPoint *sync; QSVFrame *frame; @@ -101,6 +109,9 @@ typedef struct QSVContext { mfxExtBuffer **ext_buffers; int nb_ext_buffers; + + mfxU8 payload_buffer[PAYLOAD_BUFFER_SIZE]; + Mpeg1Context mpeg_ctx; } QSVContext; static const AVCodecHWConfigInternal *const qsv_hw_configs[] = { @@ -599,6 +610,210 @@ static int qsv_export_film_grain(AVCodecContext *avctx, mfxExtAV1FilmGrainParam return 0; } #endif +static int find_start_offset(mfxU8 data[4]) +{ + if (data[0] == 0 && data[1] == 0 && data[2] == 1) + return 3; + + if (data[0] == 0 && data[1] == 0 && data[2] == 0 && data[3] == 1) + return 4; + + return 0; +} + +static int parse_sei_h264(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + H264SEIContext sei = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + start = find_start_offset(payload.Data); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else { + ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + } + + if (out) + return ff_h264_export_frame_props(avctx, &sei, NULL, out); + + return 0; +} + +static int parse_sei_hevc(AVCodecContext* avctx, QSVContext* q, QSVFrame* out) +{ + HEVCSEI sei = { 0 }; + HEVCParamSets ps = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxFrameSurface1 *surface = &out->surface; + mfxU64 ts; + int ret, has_logged = 0; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + if (!has_logged) { + has_logged = 1; + av_log(avctx, AV_LOG_VERBOSE, "-----------------------------------------\n"); + av_log(avctx, AV_LOG_VERBOSE, "Start reading SEI - payload timestamp: %llu - surface timestamp: %llu\n", ts, surface->Data.TimeStamp); + } + + if (ts != surface->Data.TimeStamp) { + av_log(avctx, AV_LOG_WARNING, "GetPayload timestamp (%llu) does not match surface timestamp: (%llu)\n", ts, surface->Data.TimeStamp); + } + + start = find_start_offset(payload.Data); + + av_log(avctx, AV_LOG_VERBOSE, "parsing SEI type: %3d Numbits %3d Start: %d\n", payload.Type, payload.NumBit, start); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + case SEI_TYPE_MASTERING_DISPLAY_COLOUR_VOLUME: + // There seems to be a bug in MSDK + payload.NumBit -= 8; + + break; + case SEI_TYPE_CONTENT_LIGHT_LEVEL_INFO: + // There seems to be a bug in MSDK + payload.NumBit = 48; + + break; + case SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35: + // There seems to be a bug in MSDK + if (payload.NumBit == 552) + payload.NumBit = 528; + break; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else { + ret = ff_hevc_decode_nal_sei(&gb, avctx, &sei, &ps, HEVC_NAL_SEI_PREFIX); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Failed to parse SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + } + + if (has_logged) { + av_log(avctx, AV_LOG_VERBOSE, "End reading SEI\n"); + } + + if (out && out->frame) + return ff_hevc_set_side_data(avctx, &sei, NULL, out->frame); + + return 0; +} + +static int parse_sei_mpeg12(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + Mpeg1Context *mpeg_ctx = &q->mpeg_ctx; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + + memset(payload.Data, 0, payload.BufSize); + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret == MFX_ERR_NOT_ENOUGH_BUFFER) { + av_log(avctx, AV_LOG_WARNING, "Warning: Insufficient buffer on GetPayload(). Size: %"PRIu64" Needed: %d\n", sizeof(q->payload_buffer), payload.BufSize); + return 0; + } + if (ret != MFX_ERR_NONE) + return ret; + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) + break; + + start = find_start_offset(payload.Data); + + start++; + + ff_mpeg_decode_user_data(avctx, mpeg_ctx, &payload.Data[start], (int)((payload.NumBit + 7) / 8) - start); + + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d start %d -> %.s\n", payload.Type, payload.NumBit, start, (char *)(&payload.Data[start])); + } + + if (!out) + return 0; + + if (mpeg_ctx->a53_buf_ref) { + + AVFrameSideData *sd = av_frame_new_side_data_from_buf(out, AV_FRAME_DATA_A53_CC, mpeg_ctx->a53_buf_ref); + if (!sd) + av_buffer_unref(&mpeg_ctx->a53_buf_ref); + mpeg_ctx->a53_buf_ref = NULL; + } + + if (mpeg_ctx->has_stereo3d) { + AVStereo3D *stereo = av_stereo3d_create_side_data(out); + if (!stereo) + return AVERROR(ENOMEM); + + *stereo = mpeg_ctx->stereo3d; + mpeg_ctx->has_stereo3d = 0; + } + + if (mpeg_ctx->has_afd) { + AVFrameSideData *sd = av_frame_new_side_data(out, AV_FRAME_DATA_AFD, 1); + if (!sd) + return AVERROR(ENOMEM); + + *sd->data = mpeg_ctx->afd; + mpeg_ctx->has_afd = 0; + } + + return 0; +} static int qsv_decode(AVCodecContext *avctx, QSVContext *q, AVFrame *frame, int *got_frame, @@ -636,6 +851,8 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, insurf, &outsurf, sync); if (ret == MFX_WRN_DEVICE_BUSY) av_usleep(500); + else if (avctx->codec_id == AV_CODEC_ID_MPEG2VIDEO) + parse_sei_mpeg12(avctx, q, NULL); } while (ret == MFX_WRN_DEVICE_BUSY || ret == MFX_ERR_MORE_SURFACE); @@ -677,6 +894,23 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, return AVERROR_BUG; } + switch (avctx->codec_id) { + case AV_CODEC_ID_MPEG2VIDEO: + ret = parse_sei_mpeg12(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_H264: + ret = parse_sei_h264(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_HEVC: + ret = parse_sei_hevc(avctx, q, out_frame); + break; + default: + ret = 0; + } + + if (ret < 0) + av_log(avctx, AV_LOG_ERROR, "Error parsing SEI data: %d\n", ret); + out_frame->queued += 1; aframe = (QSVAsyncFrame){ sync, out_frame };