From patchwork Wed Feb 10 04:57:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 25529 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E855D44BDDC for ; Wed, 10 Feb 2021 07:08:10 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BECA4689D9F; Wed, 10 Feb 2021 07:08:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga05.intel.com (unknown [192.55.52.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 385E3689C22 for ; Wed, 10 Feb 2021 07:08:02 +0200 (EET) IronPort-SDR: NDFTP1XwT+DqlzGyaMUmn8rti9KHrDkWSYj8d/aJgzRmX1/pg6Oag7MGIa67KnyShXU4vf/Otq 2YkColIw36lg== X-IronPort-AV: E=McAfee;i="6000,8403,9890"; a="266854740" X-IronPort-AV: E=Sophos;i="5.81,167,1610438400"; d="scan'208";a="266854740" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2021 21:07:51 -0800 IronPort-SDR: RAEPjZitDNgdcvuUtHwDxCky5SKJOVUJ/9GS7ppZXkXuuHP9wfHhFYqL2mfBqzv8jqaHchSFcR R02gVDthLcrA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,167,1610438400"; d="scan'208";a="379734406" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by fmsmga008.fm.intel.com with ESMTP; 09 Feb 2021 21:07:50 -0800 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 Feb 2021 12:57:49 +0800 Message-Id: <20210210045749.14624-1-yejun.guo@intel.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 01/10] dnn_backend_openvino.c: fix mismatch between ffmpeg(NHWC) and openvino(NCHW) X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Guo, Yejun --- libavfilter/dnn/dnn_backend_openvino.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index beca256390..48f5ba50be 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -250,7 +250,7 @@ static void infer_completion_callback(void *args) } } -static DNNReturnType init_model_ov(OVModel *ov_model) +static DNNReturnType init_model_ov(OVModel *ov_model, const char *input_name, const char *output_name) { OVContext *ctx = &ov_model->ctx; IEStatusCode status; @@ -276,6 +276,19 @@ static DNNReturnType init_model_ov(OVModel *ov_model) goto err; } + // The order of dims in the openvino is fixed and it is always NCHW for 4-D data. + // while we pass NHWC data from FFmpeg to openvino + status = ie_network_set_input_layout(ov_model->network, input_name, NHWC); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for input %s\n", input_name); + goto err; + } + status = ie_network_set_output_layout(ov_model->network, output_name, NHWC); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to set layout as NHWC for output %s\n", output_name); + goto err; + } + status = ie_core_load_network(ov_model->core, ov_model->network, ctx->options.device_type, &config, &ov_model->exe_network); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n"); @@ -482,7 +495,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu } if (!ov_model->exe_network) { - if (init_model_ov(ov_model) != DNN_SUCCESS) { + if (init_model_ov(ov_model, input_name, output_name) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); return DNN_ERROR; } @@ -598,7 +611,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, const char *input_n } if (!ov_model->exe_network) { - if (init_model_ov(ov_model) != DNN_SUCCESS) { + if (init_model_ov(ov_model, input_name, output_names[0]) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); return DNN_ERROR; } @@ -645,7 +658,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, const char *i } if (!ov_model->exe_network) { - if (init_model_ov(ov_model) != DNN_SUCCESS) { + if (init_model_ov(ov_model, input_name, output_names[0]) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); return DNN_ERROR; }