From patchwork Thu May 26 08:08:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 35944 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6914:b0:82:6b11:2509 with SMTP id q20csp254469pzj; Thu, 26 May 2022 01:10:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxu7OUNKTul80yEfuKTyZxVwMWUDNBFiFUJyZdybNjKLGc1Rv+7EMGo8aYLIP2KgOHqpU7Y X-Received: by 2002:a17:907:2cc7:b0:6fa:88cf:c31a with SMTP id hg7-20020a1709072cc700b006fa88cfc31amr32038006ejc.471.1653552601445; Thu, 26 May 2022 01:10:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1653552601; cv=none; d=google.com; s=arc-20160816; b=RXMZdzSqNeQ3nu4Pb9KUvkuKRITPBFhFKYDiiWxjHaOAgwGH1aSABBZSqhXyHBE532 cfaJQPGAY8HFZHrx9gr7LrNpyXXYUneDR9oxrCWZgKaOaq6wiAXEAM3cOdm258IXNJiJ hc2J2iMYeBVes6hmz+VBZ3/QGXqg230RpcKkpHmG9y/sDszaO1q8+8RDa0m3S/c8HnbL GIEzWUyEjmpR+/vxAKY1VLCd7sNh5HEuXLYooAAUFMo0EdP9tQifqmMHQzN0qDCNj3dg BBetIm2D6lE6tdZwDOzwlwqmhUkf42ll/G+XCFUo1s4j0UmrQ8EA4atX98cfyw5JKJJb Vbhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=P92F16HW0DBVQQDQZDX7nolTS7JZEXpJd/mS45DmuaQ=; b=uq0rkgFXRze/sfY8zpgViP1dPbtByCcPoPNqedUtFK1zTfBCNSmSqkFtQkIR+2lRC7 gmjYtABUSgkcKNmtsFkRXQe2wOoIfK/Sxa+kEE1UvQdw3ZBDRk8Lk389XahGgwDlw9wv XMUrjpPl7i62PN7dCfLD1LSLe4Fu5VnYCGT+8OqLpIKiEKQctakhxTCOtAntGqYopAWl 2YB0PkNQn5DLPwsequqdV6mESJDo2uMIKFLNpUov7Cjr/JcLLDXAJL6J3dVSRJ2CK5xI /hsHpjImaU6d2aBo1BaSV3QYxSP+z7XghTgYn3sfI1LbGIWySXKuyJnMSQdu6ZTwASEq MFiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=NnTuM00c; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i11-20020a170906a28b00b006fe8c8dc0f2si719270ejz.714.2022.05.26.01.10.01; Thu, 26 May 2022 01:10:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=NnTuM00c; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D44A668B59F; Thu, 26 May 2022 11:09:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D4D3B68B567 for ; Thu, 26 May 2022 11:08:58 +0300 (EEST) Received: by mail-pf1-f170.google.com with SMTP id c14so1113584pfn.2 for ; Thu, 26 May 2022 01:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:message-id:in-reply-to:references:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ro1CX39ivZCU+9wh/EltYPqCye23Nsue29SHi95XNWY=; b=NnTuM00cvsUal449HhsfXYqq5+udsfdTKOOmGuJCEEoC3WMN8MFnQDzWAMYefKLHY3 A/nmQU16irmnGlYA2IigtjCmPH+9RDujwQbXzK7Bcd7vISk5t4ue65RRlVFQtyGF9Yzk jEX6KnHAMkWp6kp4+Yk/ExgzrbNWzSaCzDL6zYqluF7MeJE5+FBR6HEBmljKnJ3aYSK/ FUzKj0FhC7qQqbNTD7o0+cDBiVHYQ3rDAQ51kg1YaW/wQcO6gOWGFz4guuGirKfSxFPv +cHM6RN3sSnKMPSXnetkv+jRcyduHVCt9zemap01G8gUeLoZ9AQ4td0dmQhIDr3tgLxc CwyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:message-id:in-reply-to:references:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ro1CX39ivZCU+9wh/EltYPqCye23Nsue29SHi95XNWY=; b=Ssr6bpM4FVmADlPtarJwB51JVwwsicmhg4vR+uzNcfQrL2nAG9Vx6bXZ+caSQlG4kF wOBimnAckCWo/UZexb2GPb2RIiW9z2Lx1h6E1afj1odyxX9czezfrGm4XagyDvSJu22q u7/r9k3PpN3pqfmD5lHr5KHfdZ3dlJnyCN3Oaj9UNypog+BR9gTYj8pjqDjLP8YeYAhx UpL0mgO8mUbukiZabHjobUu7MMbUehgQr2zA+A326xHwUzhJtORtskwvpvdPFmzWrqm2 T62KsuaCBsAyOr7wkNd3AEJxwAatSNXhlIbe4LhXvuXeDx3pP2Wg3SSFEUJIy/uFt3VX +qGw== X-Gm-Message-State: AOAM532JhitROkL9p96gk3ZZSjFIVJZ2P+cofqLMCc9nnJAlm0LPdvQV WAJYpLMcS/iAdEOyCc/X5cSF5DoWXQSPLA== X-Received: by 2002:a63:5011:0:b0:3c1:e24a:e5ea with SMTP id e17-20020a635011000000b003c1e24ae5eamr32775875pgb.359.1653552537227; Thu, 26 May 2022 01:08:57 -0700 (PDT) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id g22-20020a056a001a1600b0050dc7628148sm811681pfv.34.2022.05.26.01.08.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 May 2022 01:08:56 -0700 (PDT) From: softworkz X-Google-Original-From: softworkz Message-Id: In-Reply-To: References: Date: Thu, 26 May 2022 08:08:49 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH 6/6] avcodec/qsvdec: Implement SEI parsing for QSV decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TFqeLem1lJtM From: softworkz Signed-off-by: softworkz --- libavcodec/qsvdec.c | 233 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c index 5fc5bed4c8..7d6a491aa0 100644 --- a/libavcodec/qsvdec.c +++ b/libavcodec/qsvdec.c @@ -49,6 +49,12 @@ #include "hwconfig.h" #include "qsv.h" #include "qsv_internal.h" +#include "h264dec.h" +#include "h264_sei.h" +#include "hevcdec.h" +#include "hevc_ps.h" +#include "hevc_sei.h" +#include "mpeg12.h" static const AVRational mfx_tb = { 1, 90000 }; @@ -60,6 +66,8 @@ static const AVRational mfx_tb = { 1, 90000 }; AV_NOPTS_VALUE : pts_tb.num ? \ av_rescale_q(mfx_pts, mfx_tb, pts_tb) : mfx_pts) +#define PAYLOAD_BUFFER_SIZE 65535 + typedef struct QSVAsyncFrame { mfxSyncPoint *sync; QSVFrame *frame; @@ -101,6 +109,9 @@ typedef struct QSVContext { mfxExtBuffer **ext_buffers; int nb_ext_buffers; + + mfxU8 payload_buffer[PAYLOAD_BUFFER_SIZE]; + Mpeg1Context mpeg_ctx; } QSVContext; static const AVCodecHWConfigInternal *const qsv_hw_configs[] = { @@ -599,6 +610,208 @@ static int qsv_export_film_grain(AVCodecContext *avctx, mfxExtAV1FilmGrainParam return 0; } #endif +static int find_start_offset(mfxU8 data[4]) +{ + if (data[0] == 0 && data[1] == 0 && data[2] == 1) + return 3; + + if (data[0] == 0 && data[1] == 0 && data[2] == 0 && data[3] == 1) + return 4; + + return 0; +} + +static int parse_sei_h264(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + H264SEIContext sei = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret != MFX_ERR_NONE) { + av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); + return ret; + } + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { + break; + } + + start = find_start_offset(payload.Data); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader"); + else + ret = ff_h264_sei_decode(&sei, &gb, NULL, avctx); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "Error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + + if (out) + return ff_h264_export_frame_props(avctx, &sei, NULL, out); + + return 0; +} + +static int parse_sei_hevc(AVCodecContext* avctx, QSVContext* q, QSVFrame* out) +{ + HEVCSEI sei = { 0 }; + HEVCParamSets ps = { 0 }; + GetBitContext gb = { 0 }; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxFrameSurface1 *surface = &out->surface; + mfxU64 ts; + int ret, has_logged = 0; + + while (1) { + int start; + memset(payload.Data, 0, payload.BufSize); + + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret != MFX_ERR_NONE) { + av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); + return 0; + } + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { + break; + } + + if (!has_logged) { + has_logged = 1; + av_log(avctx, AV_LOG_VERBOSE, "-----------------------------------------\n"); + av_log(avctx, AV_LOG_VERBOSE, "Start reading SEI - payload timestamp: %llu - surface timestamp: %llu\n", ts, surface->Data.TimeStamp); + } + + if (ts != surface->Data.TimeStamp) { + av_log(avctx, AV_LOG_WARNING, "GetPayload timestamp (%llu) does not match surface timestamp: (%llu)\n", ts, surface->Data.TimeStamp); + } + + start = find_start_offset(payload.Data); + + av_log(avctx, AV_LOG_VERBOSE, "parsing SEI type: %3d Numbits %3d Start: %d\n", payload.Type, payload.NumBit, start); + + switch (payload.Type) { + case SEI_TYPE_BUFFERING_PERIOD: + case SEI_TYPE_PIC_TIMING: + continue; + case SEI_TYPE_MASTERING_DISPLAY_COLOUR_VOLUME: + // There seems to be a bug in MSDK + payload.NumBit -= 8; + + break; + case SEI_TYPE_CONTENT_LIGHT_LEVEL_INFO: + // There seems to be a bug in MSDK + payload.NumBit = 48; + + break; + case SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35: + // There seems to be a bug in MSDK + if (payload.NumBit == 552) + payload.NumBit = 528; + break; + } + + if (init_get_bits(&gb, &payload.Data[start], payload.NumBit - start * 8) < 0) + av_log(avctx, AV_LOG_ERROR, "Error initializing bitstream reader"); + else + ret = ff_hevc_decode_nal_sei(&gb, avctx, &sei, &ps, HEVC_NAL_SEI_PREFIX); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d\n", payload.Type, payload.NumBit); + } + + if (has_logged) { + av_log(avctx, AV_LOG_VERBOSE, "End reading SEI\n"); + } + + if (out && out->frame) + return ff_set_side_data(avctx, &sei, NULL, out->frame); + + return 0; +} + +static int parse_sei_mpeg12(AVCodecContext* avctx, QSVContext* q, AVFrame* out) +{ + Mpeg1Context *mpeg_ctx = &q->mpeg_ctx; + mfxPayload payload = { 0, .Data = &q->payload_buffer[0], .BufSize = sizeof(q->payload_buffer) }; + mfxU64 ts; + int ret; + + while (1) { + int start; + + memset(payload.Data, 0, payload.BufSize); + ret = MFXVideoDECODE_GetPayload(q->session, &ts, &payload); + if (ret != MFX_ERR_NONE) { + av_log(avctx, AV_LOG_WARNING, "error getting SEI payload: %d \n", ret); + return ret; + } + + if (payload.NumBit == 0 || payload.NumBit >= payload.BufSize * 8) { + break; + } + + start = find_start_offset(payload.Data); + + start++; + + ff_mpeg_decode_user_data(avctx, mpeg_ctx, &payload.Data[start], (int)((payload.NumBit + 7) / 8) - start); + + if (ret < 0) + av_log(avctx, AV_LOG_WARNING, "error parsing SEI type: %d Numbits %d error: %d\n", payload.Type, payload.NumBit, ret); + else + av_log(avctx, AV_LOG_DEBUG, "mfxPayload Type: %d Numbits %d start %d -> %.s\n", payload.Type, payload.NumBit, start, (char *)(&payload.Data[start])); + } + + if (!out) + return 0; + + if (mpeg_ctx->a53_buf_ref) { + + AVFrameSideData *sd = av_frame_new_side_data_from_buf(out, AV_FRAME_DATA_A53_CC, mpeg_ctx->a53_buf_ref); + if (!sd) + av_buffer_unref(&mpeg_ctx->a53_buf_ref); + mpeg_ctx->a53_buf_ref = NULL; + } + + if (mpeg_ctx->has_stereo3d) { + AVStereo3D *stereo = av_stereo3d_create_side_data(out); + if (!stereo) + return AVERROR(ENOMEM); + + *stereo = mpeg_ctx->stereo3d; + mpeg_ctx->has_stereo3d = 0; + } + + if (mpeg_ctx->has_afd) { + AVFrameSideData *sd = av_frame_new_side_data(out, AV_FRAME_DATA_AFD, 1); + if (!sd) + return AVERROR(ENOMEM); + + *sd->data = mpeg_ctx->afd; + mpeg_ctx->has_afd = 0; + } + + return 0; +} static int qsv_decode(AVCodecContext *avctx, QSVContext *q, AVFrame *frame, int *got_frame, @@ -636,6 +849,8 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, insurf, &outsurf, sync); if (ret == MFX_WRN_DEVICE_BUSY) av_usleep(500); + else if (avctx->codec_id == AV_CODEC_ID_MPEG1VIDEO || avctx->codec_id == AV_CODEC_ID_MPEG2VIDEO) + parse_sei_mpeg12(avctx, q, NULL); } while (ret == MFX_WRN_DEVICE_BUSY || ret == MFX_ERR_MORE_SURFACE); @@ -677,6 +892,24 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q, return AVERROR_BUG; } + switch (avctx->codec_id) { + case AV_CODEC_ID_MPEG1VIDEO: + case AV_CODEC_ID_MPEG2VIDEO: + ret = parse_sei_mpeg12(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_H264: + ret = parse_sei_h264(avctx, q, out_frame->frame); + break; + case AV_CODEC_ID_HEVC: + ret = parse_sei_hevc(avctx, q, out_frame); + break; + default: + ret = 0; + } + + if (ret < 0) + av_log(avctx, AV_LOG_ERROR, "Error parsing SEI data\n"); + out_frame->queued += 1; aframe = (QSVAsyncFrame){ sync, out_frame };