From patchwork Fri Jan 15 08:43:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 24959 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id A758F44AE1E for ; Fri, 15 Jan 2021 10:51:31 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8BFDC687EB0; Fri, 15 Jan 2021 10:51:31 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AFEA6680B67 for ; Fri, 15 Jan 2021 10:51:24 +0200 (EET) IronPort-SDR: mwb7M8OPBYDSordsy1shavfq8SPvwxcwAtUSV7d37yIjvszEvBeWK0Ed50edzq4/2sn3hPO3mB FnYDXECze2Vg== X-IronPort-AV: E=McAfee;i="6000,8403,9864"; a="157700269" X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="157700269" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Jan 2021 00:51:15 -0800 IronPort-SDR: WICWRkEeFGTKKqNzq8JkVJdJzAPocoYacq0rM2dyt/PXNMy/AnwMR0y2eURQMsutcB1oXC0J96 2GVjOKfmsxFg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="382591432" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.136]) by orsmga008.jf.intel.com with ESMTP; 15 Jan 2021 00:51:14 -0800 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 15 Jan 2021 16:43:16 +0800 Message-Id: <20210115084318.5152-1-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH V2 1/3] dnn/openvino: remove unnecessary code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_backend_openvino.c | 8 -------- 1 file changed, 8 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 5271d1caa5..8476f4fb38 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -325,14 +325,6 @@ static DNNReturnType get_input_ov(void *model, DNNData *input, const char *input return DNN_ERROR; } - // The order of dims in the openvino is fixed and it is always NCHW for 4-D data. - // while we pass NHWC data from FFmpeg to openvino - status = ie_network_set_input_layout(ov_model->network, input_name, NHWC); - if (status != OK) { - av_log(ctx, AV_LOG_ERROR, "Input \"%s\" does not match layout NHWC\n", input_name); - return DNN_ERROR; - } - input->channels = dims.dims[1]; input->height = dims.dims[2]; input->width = dims.dims[3]; From patchwork Fri Jan 15 08:43:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 24960 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 9D5E444AE1E for ; Fri, 15 Jan 2021 10:51:32 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 86EFF687FA1; Fri, 15 Jan 2021 10:51:32 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 16E23680BA4 for ; Fri, 15 Jan 2021 10:51:25 +0200 (EET) IronPort-SDR: KKCtosk9YfCgqOeOFsdzY6SDnZod2A4MXl0vo4bXvJiOzWFw5+VpZlE3fzOSKujDV31AH+bB0X DFTVfSOjlVmg== X-IronPort-AV: E=McAfee;i="6000,8403,9864"; a="157700272" X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="157700272" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Jan 2021 00:51:15 -0800 IronPort-SDR: X94Z2xCrSMnnRaRQpGgxJQ/zy4c7n7YLUNKuTuocmbyXO9vWhNhJATmrKcU4iJd5MzQFwvYUQ8 4hEMVkLwjwwQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="382591443" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.136]) by orsmga008.jf.intel.com with ESMTP; 15 Jan 2021 00:51:15 -0800 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 15 Jan 2021 16:43:17 +0800 Message-Id: <20210115084318.5152-2-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210115084318.5152-1-ting.fu@intel.com> References: <20210115084318.5152-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH V2 2/3] dnn/openvino: refine code for better model initialization X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Move openvino model/inference request creation and initialization steps from ff_dnn_load_model_ov to new function init_model_ov, for later input resize support. Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_backend_openvino.c | 196 ++++++++++++++----------- 1 file changed, 111 insertions(+), 85 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 8476f4fb38..0b125eef65 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -248,6 +248,96 @@ static void infer_completion_callback(void *args) } } +static DNNReturnType init_model_ov(OVModel *ov_model) +{ + OVContext *ctx = &ov_model->ctx; + IEStatusCode status; + ie_available_devices_t a_dev; + ie_config_t config = {NULL, NULL, NULL}; + char *all_dev_names = NULL; + + // batch size + if (ctx->options.batch_size <= 0) { + ctx->options.batch_size = 1; + } + + if (ctx->options.batch_size > 1) { + input_shapes_t input_shapes; + status = ie_network_get_input_shapes(ov_model->network, &input_shapes); + if (status != OK) + goto err; + for (int i = 0; i < input_shapes.shape_num; i++) + input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size; + status = ie_network_reshape(ov_model->network, input_shapes); + ie_network_input_shapes_free(&input_shapes); + if (status != OK) + goto err; + } + + status = ie_core_load_network(ov_model->core, ov_model->network, ctx->options.device_type, &config, &ov_model->exe_network); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n"); + status = ie_core_get_available_devices(ov_model->core, &a_dev); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n"); + goto err; + } + for (int i = 0; i < a_dev.num_devices; i++) { + APPEND_STRING(all_dev_names, a_dev.devices[i]) + } + av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all available devices are: \"%s\"\n", + ctx->options.device_type, all_dev_names); + goto err; + } + + // create infer_request for sync execution + status = ie_exec_network_create_infer_request(ov_model->exe_network, &ov_model->infer_request); + if (status != OK) + goto err; + + // create infer_requests for async execution + if (ctx->options.nireq <= 0) { + // the default value is a rough estimation + ctx->options.nireq = av_cpu_count() / 2 + 1; + } + + ov_model->request_queue = ff_safe_queue_create(); + if (!ov_model->request_queue) { + goto err; + } + + for (int i = 0; i < ctx->options.nireq; i++) { + ie_infer_request_t *request; + RequestItem *item = av_mallocz(sizeof(*item)); + if (!item) { + goto err; + } + status = ie_exec_network_create_infer_request(ov_model->exe_network, &request); + if (status != OK) { + av_freep(&item); + goto err; + } + item->infer_request = request; + item->callback.completeCallBackFunc = infer_completion_callback; + item->callback.args = item; + if (ff_safe_queue_push_back(ov_model->request_queue, item) < 0) { + av_freep(&item); + goto err; + } + } + + ov_model->task_queue = ff_queue_create(); + if (!ov_model->task_queue) { + goto err; + } + + return DNN_SUCCESS; + +err: + ff_dnn_free_model_ov(&ov_model->model); + return DNN_ERROR; +} + static DNNReturnType execute_model_ov(RequestItem *request) { IEStatusCode status; @@ -367,6 +457,13 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu in_frame->width = input_width; in_frame->height = input_height; + if (!ov_model->exe_network) { + if (init_model_ov(ov_model) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); + return DNN_ERROR; + }; + } + task.done = 0; task.do_ioproc = 0; task.async = 0; @@ -391,13 +488,10 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char *options, AVFilterContext *filter_ctx) { - char *all_dev_names = NULL; DNNModel *model = NULL; OVModel *ov_model = NULL; OVContext *ctx = NULL; IEStatusCode status; - ie_config_t config = {NULL, NULL, NULL}; - ie_available_devices_t a_dev; model = av_mallocz(sizeof(DNNModel)); if (!model){ @@ -429,88 +523,6 @@ DNNModel *ff_dnn_load_model_ov(const char *model_filename, const char *options, if (status != OK) goto err; - // batch size - if (ctx->options.batch_size <= 0) { - ctx->options.batch_size = 1; - } - - if (ctx->options.batch_size > 1) { - input_shapes_t input_shapes; - status = ie_network_get_input_shapes(ov_model->network, &input_shapes); - if (status != OK) - goto err; - for (int i = 0; i < input_shapes.shape_num; i++) - input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size; - status = ie_network_reshape(ov_model->network, input_shapes); - ie_network_input_shapes_free(&input_shapes); - if (status != OK) - goto err; - } - - status = ie_core_load_network(ov_model->core, ov_model->network, ctx->options.device_type, &config, &ov_model->exe_network); - if (status != OK) { - av_log(ctx, AV_LOG_ERROR, "Failed to init OpenVINO model\n"); - status = ie_core_get_available_devices(ov_model->core, &a_dev); - if (status != OK) { - av_log(ctx, AV_LOG_ERROR, "Failed to get available devices\n"); - goto err; - } - for (int i = 0; i < a_dev.num_devices; i++) { - APPEND_STRING(all_dev_names, a_dev.devices[i]) - } - av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all available devices are: \"%s\"\n", - ctx->options.device_type, all_dev_names); - goto err; - } - - // create infer_request for sync execution - status = ie_exec_network_create_infer_request(ov_model->exe_network, &ov_model->infer_request); - if (status != OK) - goto err; - - // create infer_requests for async execution - if (ctx->options.nireq <= 0) { - // the default value is a rough estimation - ctx->options.nireq = av_cpu_count() / 2 + 1; - } - - ov_model->request_queue = ff_safe_queue_create(); - if (!ov_model->request_queue) { - goto err; - } - - for (int i = 0; i < ctx->options.nireq; i++) { - RequestItem *item = av_mallocz(sizeof(*item)); - if (!item) { - goto err; - } - - status = ie_exec_network_create_infer_request(ov_model->exe_network, &item->infer_request); - if (status != OK) { - av_freep(&item); - goto err; - } - - item->tasks = av_malloc_array(ctx->options.batch_size, sizeof(*item->tasks)); - if (!item->tasks) { - av_freep(&item); - goto err; - } - item->task_count = 0; - - item->callback.completeCallBackFunc = infer_completion_callback; - item->callback.args = item; - if (ff_safe_queue_push_back(ov_model->request_queue, item) < 0) { - av_freep(&item); - goto err; - } - } - - ov_model->task_queue = ff_queue_create(); - if (!ov_model->task_queue) { - goto err; - } - model->get_input = &get_input_ov; model->get_output = &get_output_ov; model->options = options; @@ -554,6 +566,13 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, const char *input_n return DNN_ERROR; } + if (!ov_model->exe_network) { + if (init_model_ov(ov_model) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); + return DNN_ERROR; + }; + } + task.done = 0; task.do_ioproc = 1; task.async = 0; @@ -594,6 +613,13 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, const char *i return DNN_ERROR; } + if (!ov_model->exe_network) { + if (init_model_ov(ov_model) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n"); + return DNN_ERROR; + }; + } + task->done = 0; task->do_ioproc = 1; task->async = 1; From patchwork Fri Jan 15 08:43:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 24958 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 98DF944AE1E for ; Fri, 15 Jan 2021 10:51:25 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6E4AA680C0F; Fri, 15 Jan 2021 10:51:25 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C1FB9680AE7 for ; Fri, 15 Jan 2021 10:51:18 +0200 (EET) IronPort-SDR: 3qvDj/PJoXibFaLXeYTsjdxAYHjDzNvJx4tFv0FFmz43+ZZzG0oInsHSU43nqlwbzLlmM7NpyJ TNc2KliEV82w== X-IronPort-AV: E=McAfee;i="6000,8403,9864"; a="157700276" X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="157700276" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Jan 2021 00:51:16 -0800 IronPort-SDR: ClQXXWPiT5aC/0v5tq1/0FN4pDxGO3Dha98sqI4pHQSGzXKdllSpPKZj1LWL7rxGVLMU7Vszvg COh8Dl2ZQR8g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,349,1602572400"; d="scan'208";a="382591445" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.136]) by orsmga008.jf.intel.com with ESMTP; 15 Jan 2021 00:51:16 -0800 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 15 Jan 2021 16:43:18 +0800 Message-Id: <20210115084318.5152-3-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210115084318.5152-1-ting.fu@intel.com> References: <20210115084318.5152-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH V2 3/3] dnn/openvino: support model input resize X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" OpenVINO APIs require specify input size to run the model, while some OpenVINO model does accept different input size. To enable this feature adding input_resizable option here for easier use. Setting bool variable input_resizable to specify if the input can be resizable or not. input_resizable = 1 means support input resize, aka accept different input size. input_resizable = 0 (default) means do not support input resize. Please make sure the inference model does accept different input size before use this option, otherwise the inference engine may report error(s). eg: ./ffmpeg -i video_name.mp4 -vf dnn_processing=dnn_backend=openvino:\ model=model_name.xml:input=input_name:output=output_name:\ options=device=CPU\&input_resizable=1 -y output_video_name.mp4 Signed-off-by: Ting Fu --- V2: rebase to latest code libavfilter/dnn/dnn_backend_openvino.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 0b125eef65..1664ff5268 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -38,6 +38,7 @@ typedef struct OVOptions{ char *device_type; int nireq; int batch_size; + int input_resizable; } OVOptions; typedef struct OVContext { @@ -86,6 +87,7 @@ static const AVOption dnn_openvino_options[] = { { "device", "device to run model", OFFSET(options.device_type), AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS }, { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, { "batch_size", "batch size per request", OFFSET(options.batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, + { "input_resizable", "can input be resizable or not", OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL } }; @@ -393,6 +395,7 @@ static DNNReturnType get_input_ov(void *model, DNNData *input, const char *input size_t model_input_count = 0; dimensions_t dims; precision_e precision; + int input_resizable = ctx->options.input_resizable; status = ie_network_get_inputs_number(ov_model->network, &model_input_count); if (status != OK) { @@ -416,8 +419,8 @@ static DNNReturnType get_input_ov(void *model, DNNData *input, const char *input } input->channels = dims.dims[1]; - input->height = dims.dims[2]; - input->width = dims.dims[3]; + input->height = input_resizable ? -1 : dims.dims[2]; + input->width = input_resizable ? -1 : dims.dims[3]; input->dt = precision_to_datatype(precision); return DNN_SUCCESS; } else { @@ -443,6 +446,8 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu AVFrame *in_frame = av_frame_alloc(); AVFrame *out_frame = NULL; TaskItem *ptask = &task; + IEStatusCode status; + input_shapes_t input_shapes; if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); @@ -457,6 +462,18 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu in_frame->width = input_width; in_frame->height = input_height; + if (ctx->options.input_resizable) { + status = ie_network_get_input_shapes(ov_model->network, &input_shapes); + input_shapes.shapes->shape.dims[2] = input_height; + input_shapes.shapes->shape.dims[3] = input_width; + status |= ie_network_reshape(ov_model->network, input_shapes); + ie_network_input_shapes_free(&input_shapes); + if (status != OK) { + av_log(ctx, AV_LOG_ERROR, "Failed to reshape input size for %s\n", input_name); + return DNN_ERROR; + } + } + if (!ov_model->exe_network) { if (init_model_ov(ov_model) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed init OpenVINO exectuable network or inference request\n");