From patchwork Thu Dec 23 02:26:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wenbin Chen X-Patchwork-Id: 32847 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp6900328iog; Wed, 22 Dec 2021 18:27:04 -0800 (PST) X-Google-Smtp-Source: ABdhPJyrZlBcKNbpiX8nmb3404x9dXz9qTPXzhcHvSLqP2hoxaC8Vu8M6qplPyBpDd6uRefP/e1+ X-Received: by 2002:a17:906:b759:: with SMTP id fx25mr364195ejb.753.1640226424694; Wed, 22 Dec 2021 18:27:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1640226424; cv=none; d=google.com; s=arc-20160816; b=UCG+nMdMrbOugBa1sjF0IOkIUi679d5tAK04/P4d54CgpopLBVqwVdOea0n/I/eTxH hy9XQVq9zG84tfYJa2xxJitKEJS9iNwhIQrBpxtLp+RacJphbm6UzqBo2Wy9mJdRIm7l YNNMXc7lDRjRprdPd18Wn7ACF3Jde8ZsqUBkER1voaEiFGCqPZBbB4UXZb5Pc4Xpoaks gYBf20bna8Y3DPMA1zf78XlfSZf40yPnIxyr0snrx3VFpl7qljZ5RUNZxlv0lQDVBWSu cv3yajkLpBI39Pv7H4cKalW6JiPkM3tSK0jcb5yVHVuPrPQtoWxxaTXQX6haWJAeLFdA pauA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=elSG+rOr35XDgsseJqXbXM9fOAmgU6QlRdHXSertJZU=; b=HBwalTz2g97wHHUb4BXKAlYn969fHVsv5h8rLR7BNMzueVJ5kEZD3snaweuuwQvqxp Jy70aOxr+GMEFP2jon+HBcwLw11sw3HfxcKV4e5TovCR1FuUOo5GnPiymY7RzrAX22cZ PJMwgFmRwyRZBgetxWG9O1SLB8R6+K0zcJ6D4IK+3VJVOazWRtaWl8q4LzIX39AaTQHf oNQYYZTnOMn/sJ170CPOO8h0xjeHnyxCIzaOlgQlqfsH4DMY4WlQ/Y7tFAUg9YOt7mGN Cr/cRsTic4b+yus+HaiH00GXp+3rl1ln/Jamn9qQckl3Kqp83fhI2gqsFTq2pMPaYqV2 mh2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=dy3r3z+I; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o5si1662047edq.585.2021.12.22.18.27.03; Wed, 22 Dec 2021 18:27:04 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=dy3r3z+I; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AE3FE68B0FB; Thu, 23 Dec 2021 04:26:59 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CB7CC68AF88 for ; Thu, 23 Dec 2021 04:26:52 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1640226418; x=1671762418; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=gIq93eESli02nvFkUN1kQvfXGAyfI0yv0Ig3UIz04is=; b=dy3r3z+IzSrxCKn1pPaO9THt9m7UNGG6JoBOkHr+GOE7E8MwjzVz79nz SCvpsQNnawb4n+bCoFNOxy2GSL076eUOx3DzeElbqHVybchcZ7/i+7Lu4 6zWldtoB84uIz/frrOgMimDDrn4ZFTcwmPd1S5YFirV6iDxjqmG/iMW7e iwDPE7rXHm3LxKJrg+5ZU/O70/Stnh8+mP5/XngPiPcTjsnLMuqQjBKMZ rff8z/1cW7don7mM7r85SYPk9dDHHMrlQli4Koe/HgJSFqT1pcUgJ9UEg QjBhkuEI+o9ObRv+l6huDD0RzNgEagLg+x8HEf20zATbw0OL3EzMZLx8T A==; X-IronPort-AV: E=McAfee;i="6200,9189,10206"; a="327048749" X-IronPort-AV: E=Sophos;i="5.88,228,1635231600"; d="scan'208";a="327048749" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Dec 2021 18:26:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,228,1635231600"; d="scan'208";a="468372768" Received: from chenwenbin-z390-aorus-ultra.sh.intel.com ([10.239.35.110]) by orsmga006.jf.intel.com with ESMTP; 22 Dec 2021 18:26:48 -0800 From: Wenbin Chen To: ffmpeg-devel@ffmpeg.org Date: Thu, 23 Dec 2021 10:26:37 +0800 Message-Id: <20211223022637.1290934-1-wenbin.chen@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2] libavutil/hwcontext_qsv: clean padding when upload qsv frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Wenbin Chen Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: x+EQPUA37Sdk Fix #7830 When we upload a frame that is not padded as MSDK requires, we create a new AVFrame to copy data. The frame's padding data is uninitialized so it brings run to run problem. For example, If we run the following command serveral times we will get different outputs. ffmpeg -init_hw_device qsv=qsv:hw -qsv_device /dev/dri/renderD128 \ -filter_hw_device qsv -f rawvideo -s 192x200 -pix_fmt p010 \ -i 192x200_P010.yuv -vf "format=nv12,hwupload=extra_hw_frames=16" \ -c:v hevc_qsv output.265 According to https://github.com/Intel-Media-SDK/MediaSDK/blob/master/doc/mediasdk-man.md#encoding-procedures "Note: It is the application's responsibility to fill pixels outside of crop window when it is smaller than frame to be encoded. Especially in cases when crops are not aligned to minimum coding block size (16 for AVC, 8 for HEVC and VP9)" I add a function to fill padding area with border pixel to fix this run2run problem, and also move the new AVFrame to global structure to reduce redundant allocation operation to increase preformance. Signed-off-by: Wenbin Chen --- libavutil/hwcontext_qsv.c | 96 +++++++++++++++++++++++++++++++++------ 1 file changed, 83 insertions(+), 13 deletions(-) diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c index a5d154a24d..853fb7f60d 100644 --- a/libavutil/hwcontext_qsv.c +++ b/libavutil/hwcontext_qsv.c @@ -47,6 +47,7 @@ #include "pixfmt.h" #include "pixdesc.h" #include "time.h" +#include "imgutils.h" #define QSV_VERSION_ATLEAST(MAJOR, MINOR) \ (MFX_VERSION_MAJOR > (MAJOR) || \ @@ -90,6 +91,7 @@ typedef struct QSVFramesContext { mfxExtOpaqueSurfaceAlloc opaque_alloc; mfxExtBuffer *ext_buffers[1]; + AVFrame realigned_tmp_frame; } QSVFramesContext; static const struct { @@ -137,6 +139,54 @@ static uint32_t qsv_get_d3d11va_bind_flags(int mem_type) } #endif +static int qsv_fill_border(AVFrame *dst, const AVFrame *src) +{ + const AVPixFmtDescriptor *desc; + int i, planes_nb = 0; + if (dst->format != src->format) + return AVERROR(EINVAL); + + desc = av_pix_fmt_desc_get(dst->format); + + for (i = 0; i < desc->nb_components; i++) + planes_nb = FFMAX(planes_nb, desc->comp[i].plane + 1); + + for (i = 0; i < planes_nb; i++) { + int sheight, dheight, y; + ptrdiff_t swidth = av_image_get_linesize(src->format, + src->width, + i); + ptrdiff_t dwidth = av_image_get_linesize(dst->format, + dst->width, + i); + const AVComponentDescriptor comp = desc->comp[i]; + if (swidth < 0 || dwidth < 0) { + av_log(NULL, AV_LOG_ERROR, "av_image_get_linesize failed\n"); + return AVERROR(EINVAL); + } + sheight = src->height; + dheight = dst->height; + if (i) { + sheight = AV_CEIL_RSHIFT(src->height, desc->log2_chroma_h); + dheight = AV_CEIL_RSHIFT(dst->height, desc->log2_chroma_h); + } + //fill right padding + for (y = 0; y < sheight; y++) { + void *line_ptr = dst->data[i] + y*dst->linesize[i] + swidth; + av_memcpy_backptr(line_ptr, + comp.depth > 8 ? 2 : 1, + dwidth - swidth); + } + //fill bottom padding + for (y = sheight; y < dheight; y++) { + memcpy(dst->data[i]+y*dst->linesize[i], + dst->data[i]+(sheight-1)*dst->linesize[i], + dwidth); + } + } + return 0; +} + static int qsv_device_init(AVHWDeviceContext *ctx) { AVQSVDeviceContext *hwctx = ctx->hwctx; @@ -220,6 +270,7 @@ static void qsv_frames_uninit(AVHWFramesContext *ctx) av_freep(&s->surface_ptrs); av_freep(&s->surfaces_internal); av_freep(&s->handle_pairs_internal); + av_frame_unref(&s->realigned_tmp_frame); av_buffer_unref(&s->child_frames_ref); } @@ -1014,12 +1065,13 @@ static int qsv_transfer_data_to(AVHWFramesContext *ctx, AVFrame *dst, QSVFramesContext *s = ctx->internal->priv; mfxFrameSurface1 in = {{ 0 }}; mfxFrameSurface1 *out = (mfxFrameSurface1*)dst->data[3]; + mfxFrameInfo tmp_info; mfxSyncPoint sync = NULL; mfxStatus err; int ret = 0; /* make a copy if the input is not padded as libmfx requires */ - AVFrame tmp_frame; + AVFrame *tmp_frame = &s->realigned_tmp_frame; const AVFrame *src_frame; int realigned = 0; @@ -1048,24 +1100,40 @@ static int qsv_transfer_data_to(AVHWFramesContext *ctx, AVFrame *dst, if (ret < 0) return ret; + /* According to MSDK spec for mfxframeinfo, "Width must be a multiple of 16. + * Height must be a multiple of 16 for progressive frame sequence and a + * multiple of 32 otherwise.", so allign all frames to 16 before uploading. */ if (src->height & 15 || src->linesize[0] & 15) { realigned = 1; - memset(&tmp_frame, 0, sizeof(tmp_frame)); - tmp_frame.format = src->format; - tmp_frame.width = FFALIGN(src->width, 16); - tmp_frame.height = FFALIGN(src->height, 16); - ret = av_frame_get_buffer(&tmp_frame, 0); - if (ret < 0) + if (tmp_frame->format != src->format || + tmp_frame->width != FFALIGN(src->width, 16) || + tmp_frame->height != FFALIGN(src->height, 16)) { + av_frame_unref(tmp_frame); + + tmp_frame->format = src->format; + tmp_frame->width = FFALIGN(src->width, 16); + tmp_frame->height = FFALIGN(src->height, 16); + ret = av_frame_get_buffer(tmp_frame, 0); + if (ret < 0) + return ret; + } + ret = av_frame_copy(tmp_frame, src); + if (ret < 0) { + av_frame_unref(tmp_frame); return ret; - - ret = av_frame_copy(&tmp_frame, src); + } + ret = qsv_fill_border(tmp_frame, src); if (ret < 0) { - av_frame_unref(&tmp_frame); + av_frame_unref(tmp_frame); return ret; } + + tmp_info = out->Info; + out->Info.CropW = FFMIN(out->Info.Width, tmp_frame->width); + out->Info.CropH = FFMIN(out->Info.Height, tmp_frame->height); } - src_frame = realigned ? &tmp_frame : src; + src_frame = realigned ? tmp_frame : src; if (!s->session_upload) { if (s->child_frames_ref) @@ -1097,8 +1165,10 @@ static int qsv_transfer_data_to(AVHWFramesContext *ctx, AVFrame *dst, return AVERROR_UNKNOWN; } - if (realigned) - av_frame_unref(&tmp_frame); + if (realigned) { + out->Info.CropW = tmp_info.CropW; + out->Info.CropH = tmp_info.CropH; + } return 0; }