From patchwork Fri Jul 24 02:39:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiang, Haihao" X-Patchwork-Id: 21243 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 109A8447CCB for ; Fri, 24 Jul 2020 05:39:44 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C83AE68B66F; Fri, 24 Jul 2020 05:39:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3FAD468ABC2 for ; Fri, 24 Jul 2020 05:39:37 +0300 (EEST) IronPort-SDR: j5Bk2Ovfw0nGhsZSLiuFyVRADMi8ZxqNMsLSDn3l5ETGNuqfgimsM6VUJkcrqBF0aT1B/h3r0i bdQAKBo7mOEw== X-IronPort-AV: E=McAfee;i="6000,8403,9691"; a="212185653" X-IronPort-AV: E=Sophos;i="5.75,389,1589266800"; d="scan'208";a="212185653" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Jul 2020 19:39:34 -0700 IronPort-SDR: 6T7w4kZv1NQfaOmUsP4DoNhggVigWvn6yd1TQ0ghPMqCMftEvyEzaPTspKdCbei5jghfUMuwFu J+j9G9ybINCg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,389,1589266800"; d="scan'208";a="288852284" Received: from xhh-tgl64.sh.intel.com ([10.239.159.152]) by orsmga006.jf.intel.com with ESMTP; 23 Jul 2020 19:39:33 -0700 From: Haihao Xiang To: ffmpeg-devel@ffmpeg.org Date: Fri, 24 Jul 2020 10:39:09 +0800 Message-Id: <20200724023909.1156568-1-haihao.xiang@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] hwcontext_vaapi: remove duplicate formats from sw_format list X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Haihao Xiang Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" hwcontext_vaapi maps different VA fourcc to the same pix_fmt for U/V plane swap cases, however duplicate formats are not expected in sw_format list when merging formats. For example: ffmpeg -loglevel debug -init_hw_device vaapi -filter_hw_device vaapi0 \ -f lavfi -i smptebars -vf \ "hwupload=derive_device=vaapi,scale_vaapi,hwdownload,format=yuv420p" \ -vframes 1 -f null - Without this fix, an auto scaler is required for the above command Duplicate formats in ff_merge_formats detected [auto_scaler_0 @ 0x560df58f4550] Setting 'flags' to value 'bicubic' [auto_scaler_0 @ 0x560df58f4550] w:iw h:ih flags:'bicubic' interl:0 [Parsed_hwupload_0 @ 0x560df58f0ec0] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_hwupload_0' Signed-off-by: Haihao Xiang --- libavutil/hwcontext_vaapi.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c index b31cf95850..fb9be19647 100644 --- a/libavutil/hwcontext_vaapi.c +++ b/libavutil/hwcontext_vaapi.c @@ -268,14 +268,24 @@ static int vaapi_frames_get_constraints(AVHWDeviceContext *hwdev, } for (i = j = 0; i < attr_count; i++) { + int k; + if (attr_list[i].type != VASurfaceAttribPixelFormat) continue; fourcc = attr_list[i].value.value.i; pix_fmt = vaapi_pix_fmt_from_fourcc(fourcc); - if (pix_fmt != AV_PIX_FMT_NONE) + + if (pix_fmt == AV_PIX_FMT_NONE) + continue; + + for (k = 0; k < j; k++) { + if (constraints->valid_sw_formats[k] == pix_fmt) + break; + } + + if (k == j) constraints->valid_sw_formats[j++] = pix_fmt; } - av_assert0(j == pix_fmt_count); constraints->valid_sw_formats[j] = AV_PIX_FMT_NONE; } } else { @@ -287,9 +297,19 @@ static int vaapi_frames_get_constraints(AVHWDeviceContext *hwdev, err = AVERROR(ENOMEM); goto fail; } - for (i = 0; i < ctx->nb_formats; i++) - constraints->valid_sw_formats[i] = ctx->formats[i].pix_fmt; - constraints->valid_sw_formats[i] = AV_PIX_FMT_NONE; + for (i = j = 0; i < ctx->nb_formats; i++) { + int k; + + for (k = 0; k < j; k++) { + if (constraints->valid_sw_formats[k] == ctx->formats[i].pix_fmt) + break; + } + + if (k == j) + constraints->valid_sw_formats[j++] = ctx->formats[i].pix_fmt; + } + + constraints->valid_sw_formats[j] = AV_PIX_FMT_NONE; } constraints->valid_hw_formats = av_malloc_array(2, sizeof(pix_fmt));