From patchwork Sun Aug 8 10:55:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29348 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981310ioh; Sun, 8 Aug 2021 03:57:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJziY+D0zVyoUi2TZ4LZqQQHphj8gdAJTVg3tPr281O7XsBaKIIQY9QHDnympmskgV37HxxY X-Received: by 2002:aa7:da02:: with SMTP id r2mr23566305eds.249.1628420246772; Sun, 08 Aug 2021 03:57:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420246; cv=none; d=google.com; s=arc-20160816; b=Xi2iZ3kqKQH/sbjoFawz0H8o/ZWIAy1e5SX7cP+/PNE8Lmf1wJWs7Bj0Q39+loYGE6 iuAb/nRlD5sFjQqDzwfpGPnYhjEkKDo49pvWbq5sui0c8gto0KXPXqx22WLzTzB9pM9u qq3+3FkVXvEFkWPfPkvcQoCRczQGiEieMN9ZYNE0n6hxH4ilJuKsAFo4PZtbPLnI7w+R NSDeC6GM7xd8dKpxcrkv7rQ+DpjDAcG370/kzYA7fo1+uWhrl4OAep/X6cR3EEkMY6Oy J3y4ST2d2Z+7ZCHotCsiQcWQeDMm/oR+K4aQYWb61k4suDJ0xxD9KR3DyVpx5XLYjURB GS1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=OgUuaP8xPpRGK6OrUayodP2ww1RTim5e3wbbMR4gbAE=; b=WP753+pASatWdZr7BHOeuH4wI3pYzAn7DAyTL2uDwHOgn2J9zcyr0O63gENJecCA9v Fy/hCnrImpITjh3VVfzU0VyvWfDqAHFGQKP+B8i3cXxZUeU/v5WDVF7ZweXOHECFSq4W td6hgQD7Vv+PH4Ex78huN8blnuBGbOqV8pq+M6e+5N7nFekqM/WYda9zSwW8iLQ9Ag/D sckBPtTHTfoxec41op3w/nMokhqQWQVQBExq8LhIDKNCi1ScMTuqPJH+dmbaXFEDPzQ6 e5a9A+uHsykt9Kx1LGZxx26qUe0np07URuvEXdgL6tuXznN3DlTWq7vZA+KVQHRyz6wX 6q9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=k43iN1jm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cq4si14883876edb.74.2021.08.08.03.57.26; Sun, 08 Aug 2021 03:57:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=k43iN1jm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3E6FD68A52D; Sun, 8 Aug 2021 13:56:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0F91168A441 for ; Sun, 8 Aug 2021 13:56:20 +0300 (EEST) Received: by mail-pl1-f182.google.com with SMTP id f3so2716906plg.3 for ; Sun, 08 Aug 2021 03:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N+koUDzWtuSowqPYVU7R/lvjG5E1OjXVK6thHCxRP2E=; b=k43iN1jmjOPJJ4lc7zdC4R+qllD5EKXXOBnXvJPdct5QI7dMUwkhAAoUwOkM3uZUGp fsUS1uZOY6sBZt7N6/mzStQPSbP+Zfdm04yPIopfX3Lf4xSghve1Pgrr8eJYaLLHIJfF RmNg+S5C8Kto3akIiFy7sZtfr16F/QvBI6eaUUUXLC8r1mEmwE9O5HTVL/C4iMlEVCfa vDJRBhUAn070F+Z7c5M7Ub/8RGVWOKSJSdD7g0Uy8WMQOCGg1oFiEcrGBaFxeLI48kVJ B+DckISkbvtebm8HHwOGVmeWt53ugiXTahvUzhHhCYSg+aYOhTaknYOnypbwkb2CToY4 RPpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N+koUDzWtuSowqPYVU7R/lvjG5E1OjXVK6thHCxRP2E=; b=TMAT2o8dbgtagNY9tN2zudOB9EOTxVBFvGW5j2P6QzcaiWV4Xclicvk+bLKbGJqCt3 xFDGq4+Gm8G8CGR0zrfe1lDUzQJVbhr8BrSQhbdNIfo3EZpxETwAmrvdx0CZ1cIcCGIj ZsOOcWPjjUU9R71v9+ta8wDl5jnHG8MExQYdKHjCrTD9kcVd5Zlt5hAXrtt9YOFuCHUX QTt8kJMt8tIAvFqZC+61IBcdMlcK5wApz5cljw/iuj7U0MeVBdMZcyjwf8XBCai2uCsk vH0F9OTOAusRRaiYUvj8+x6n2rMVrutjbcr/YuX9BXcBZirsWOGK+jGk1urFoBJF43ao Agtw== X-Gm-Message-State: AOAM5304Md9pd13Q+/3CLRzSBqWJRk4eIAEFyeDnNzROJTwUgf6+m3eT 86v2ysBw2ksuT5kvaGHhj9WCtEESPEJgdQ== X-Received: by 2002:a17:90b:3810:: with SMTP id mq16mr22236301pjb.168.1628420178273; Sun, 08 Aug 2021 03:56:18 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:17 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:37 +0530 Message-Id: <20210808105539.18822-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 7/9] [GSoC] lavfi/dnn: Extract Common Parts from get_output functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: z1bS6HnGOzZD The frame allocation and filling the TaskItem with execution parameters is common in the three backends. This commit shifts this logic to dnn_backend_common. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 26 +++++++++++++ libavfilter/dnn/dnn_backend_common.h | 16 ++++++++ libavfilter/dnn/dnn_backend_openvino.c | 51 +++++++++----------------- libavfilter/dnn/dnn_backend_tf.c | 42 +++++++-------------- 4 files changed, 72 insertions(+), 63 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index df4bab85e0..470fffa2ae 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -142,3 +142,29 @@ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **i return DAST_SUCCESS; } + +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx) +{ + AVFrame *in_frame = NULL; + AVFrame *out_frame = NULL; + + in_frame = av_frame_alloc(); + if (!in_frame) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + return DNN_ERROR; + } + + out_frame = av_frame_alloc(); + if (!out_frame) { + av_frame_free(&in_frame); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); + return DNN_ERROR; + } + + in_frame->width = input_width; + in_frame->height = input_height; + exec_params->in_frame = in_frame; + exec_params->out_frame = out_frame; + + return ff_dnn_fill_task(task, exec_params, backend_model, 0, 0); +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index e1d678b230..604c1d3bd7 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -137,4 +137,20 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ */ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); +/** + * Allocate input and output frames and fill the Task + * with execution parameters. + * + * @param task pointer to the allocated task + * @param exec_params pointer to execution parameters + * @param backend_model void pointer to the backend model + * @param input_height height of input frame + * @param input_width width of input frame + * @param ctx pointer to the backend context + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if allocation fails + */ +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 47cbd48ba2..c825e70c82 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -640,10 +640,15 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu OVContext *ctx = &ov_model->ctx; TaskItem task; OVRequestItem *request; - AVFrame *in_frame = NULL; - AVFrame *out_frame = NULL; IEStatusCode status; input_shapes_t input_shapes; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; if (ov_model->model->func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); @@ -669,51 +674,29 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu } } - in_frame = av_frame_alloc(); - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, ov_model, input_height, input_width, ctx) != DNN_SUCCESS) { return DNN_ERROR; } - in_frame->width = input_width; - in_frame->height = input_height; - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; - } - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.nb_output = 1; - task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } ret = execute_model_ov(request, ov_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; - - av_frame_free(&out_frame); - av_frame_free(&in_frame); + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; +err: + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 935a2618a1..3dfadf888a 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -310,36 +310,20 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu DNNReturnType ret; TFModel *tf_model = model; TFContext *ctx = &tf_model->ctx; - AVFrame *in_frame = av_frame_alloc(); - AVFrame *out_frame = NULL; TaskItem task; TFRequestItem *request; - - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - ret = DNN_ERROR; - goto err; - } - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - ret = DNN_ERROR; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; + + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, tf_model, input_height, input_width, ctx) != DNN_SUCCESS) { goto err; } - in_frame->width = input_width; - in_frame->height = input_height; - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.model = tf_model; - task.nb_output = 1; - if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); ret = DNN_ERROR; @@ -354,12 +338,12 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu } ret = execute_model_tf(request, tf_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; err: - av_frame_free(&out_frame); - av_frame_free(&in_frame); + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; }