From patchwork Mon Jan 11 02:20:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 24893 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 33AD744BA65 for ; Mon, 11 Jan 2021 04:28:49 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1F0DE68AB5B; Mon, 11 Jan 2021 04:28:49 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8CB9268A774 for ; Mon, 11 Jan 2021 04:28:41 +0200 (EET) IronPort-SDR: /prGuM5ytpMXx7aWaleE9BSBSBLIXRK2ZE8Nsk4kBN8Wl7hxdFpbsVV7BEvcjGkwoYNkgpcFdM jTcmefs2CJaA== X-IronPort-AV: E=McAfee;i="6000,8403,9860"; a="239337965" X-IronPort-AV: E=Sophos;i="5.79,337,1602572400"; d="scan'208";a="239337965" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jan 2021 18:28:33 -0800 IronPort-SDR: Eh0GvlhTZKrtr07mQfvLM7bqrM7HGwD3Uhl9tsixM+IBTD6CsjG6YuU4yVESGqjdriU0xckTi5 CNuh5Z34FmTA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,337,1602572400"; d="scan'208";a="380848671" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.136]) by orsmga008.jf.intel.com with ESMTP; 10 Jan 2021 18:28:32 -0800 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Mon, 11 Jan 2021 10:20:32 +0800 Message-Id: <20210111022032.22334-3-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210111022032.22334-1-ting.fu@intel.com> References: <20210111022032.22334-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 3/3] dnn/openvino: support model input resize X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" OpenVINO APIs require specify input size to run the model, while some OpenVINO model does accept different input size. To enable this feature adding input_resizable option here for easier use. Setting bool variable input_resizable to specify if the input can be resizable or not. input_resizable = 1 means support input resize, aka accept different input size. input_resizable = 0 (default) means do not support input resize. Please make sure the inference model does accept different input size before use this option, otherwise the inference engine may report error(s). eg: ./ffmpeg -i video_name.mp4 -vf dnn_processing=dnn_backend=openvino:\ model=model_name.xml:input=input_name:output=output_name:\ options=device=CPU\&input_resizable=1 -y output_video_name.mp4 Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_backend_openvino.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index d6e0593a0b..65d74702ff 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -37,6 +37,7 @@ typedef struct OVOptions{ char *device_type; int nireq; + int input_resizable; } OVOptions; typedef struct OVContext { @@ -83,6 +84,7 @@ typedef struct RequestItem { static const AVOption dnn_openvino_options[] = { { "device", "device to run model", OFFSET(options.device_type), AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS }, { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + { "input_resizable", "can input be resizable or not", OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL } }; @@ -334,6 +336,7 @@ static DNNReturnType get_input_ov(void *model, DNNData *input, const char *input size_t model_input_count = 0; dimensions_t dims; precision_e precision; + int input_resizable = ctx->options.input_resizable; status = ie_network_get_inputs_number(ov_model->network, &model_input_count); if (status != OK) { @@ -357,8 +360,8 @@ static DNNReturnType get_input_ov(void *model, DNNData *input, const char *input } input->channels = dims.dims[1]; - input->height = dims.dims[2]; - input->width = dims.dims[3]; + input->height = input_resizable ? -1 : dims.dims[2]; + input->width = input_resizable ? -1 : dims.dims[3]; input->dt = precision_to_datatype(precision); return DNN_SUCCESS; } else { @@ -383,6 +386,8 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu RequestItem request; AVFrame *in_frame = av_frame_alloc(); AVFrame *out_frame = NULL; + IEStatusCode status; + input_shapes_t input_shapes; if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); @@ -397,6 +402,18 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu in_frame->width = input_width; in_frame->height = input_height; + if (ctx->options.input_resizable) { + status = ie_network_get_input_shapes(ov_model->network, &input_shapes); + input_shapes.shapes->shape.dims[2] = input_height; + input_shapes.shapes->shape.dims[3] = input_width; + status |= ie_network_reshape(ov_model->network, input_shapes); + ie_network_input_shapes_free(&input_shapes); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to reshape input size for %s\n", input_name); + return DNN_ERROR; + } + } + if (!ov_model->exe_network) { if (init_model_ov(ov_model) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n");