From patchwork Sat Jun 5 18:08:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28104 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp2155688iof; Sat, 5 Jun 2021 11:38:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxd/YVd2sezQWsuJR3HVLWZN5QhQsJTh2/mqdVF/d+pM8V1iZVTgvnrzxY70m77Iw+mg+/O X-Received: by 2002:a17:906:3912:: with SMTP id f18mr10128730eje.161.1622918308094; Sat, 05 Jun 2021 11:38:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622918308; cv=none; d=google.com; s=arc-20160816; b=NYr8qEbJJ52JTC7k/v5z4IBPb59pw1V3iqTDGOp+VOjlThKFX7ykOGzjJaBHRPVgc/ b2csv6HFQsFkCUaN0D6W91F0JCMVxVECApvd1WNcDHQRi/jmKTi5px0L/s6I7O1tOHO/ nO5b7G4v5iMHVbhbYhFHmlaTgrgS91jF3fZVBCJDp0jFN/qygi5Ukn1VZ/urIZJmcw6w PWigmpNT0AyF9Wjbfa6IwnFh902K8rFieCMpgsvVxVf94mmYaY9/GD5rCw4Ws9h7jsOB 9qtOPHuKCLD3pEnyV47lVkR+xQdlOnuFAVRYuDgJIldnsBSkzeAhKEVq+HdxKMzciLBP I12g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=GEo/0/1XrAdrKdht1l/eg+5SqgjoYss5ufrRkgIYRus=; b=ZjRUCJxl/ywEcuLkNavlDcqqhDIcT1hA4tc4Bkhqba8hWj7HWKNCd0mgDQpSixUcyy KboOxPGIOAUY0Zso2mreP2EsVYO7R8S/4abZpK2OsbVgqD5hrgD/vZlaG+Jd4sJmASe3 ImQwkfii/oG2DzHBe3b2Xdh4cnfkGzTc1fk4tn0JJ/qa9FsxEl1L4cYEu5e+JK/cvtWG cM+htOnxkHJv3SbNzdId4RCkAcnK6OlzvHdMO9Jb9jswxBmsGqostYHTlx6qgrS3ApQq qh0ClgeHA/915yAlcOYpqRdBqDbXpUV7HnbrI2IWCTlUu2tsU0FF4G2eZ6NkBc/nVRwt mnmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=M5yk9kRy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m4si9090037eja.301.2021.06.05.11.38.27; Sat, 05 Jun 2021 11:38:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=M5yk9kRy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1F2C868A62B; Sat, 5 Jun 2021 21:38:24 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 87EC068A4B0 for ; Sat, 5 Jun 2021 21:38:17 +0300 (EEST) Received: by mail-oi1-f174.google.com with SMTP id m137so9537009oig.6 for ; Sat, 05 Jun 2021 11:38:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=M5yk9kRyEvgntDvk1tsLJtbv4RkTaww9PZBm/rF5a8m0TPAN1ctplkTSRcqjKsYwOP a1YsDEMhOCtRA3AHWzJqrh2hgjLpn3gEibrHdBpNP1A0m/xzneWfw6JeobR+ar4Ql5m+ UfCogFdO6W9PtZncJHLK7OdnMlK4xqoYr9MGTHHsJMzIKaXLf7gCaaU2NY0+tGpQSYGR P1sxJyWqDbhhmptoPoGhBdlyBRs1srfCG6A2MbL5Y8GjPKfQ07la7JB3I+JXIfoCoEa4 V+VGb0BcMSDcGgjN863+JWS0oi3ojSOEuH/pJn7JRii5K9Bkub6EaaNelIQOAYFCDtW8 HyTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=Xn/uJLeQuopAq/XGO9snEfjw7DE+NnIuyWe+wRZNYSJLnPl4F6LZXt12Wx/rhymFW7 jAsFyxtB5UxinxCWG/4W+PHGaR3M9r3wyHfTvtx6wH+4m/SASEZIfA4cp0doRd1LyPhH Bxc7dxjz8wxqcx/bo3+f5xWySo0qPpglsPCQfF3PAeyFfUNa6SxcfAAP0j0byTo8YaZ3 oMNwMWz/+gWzIOrDNzyan0ZL2l7OtvNp1VAzwY2w0V2/cvwbctUG0oub907iPVwJYTg9 wZ+i5I0kfs9kU+if5MjvVf2XMhyUMv0Hs7jpg9SjnpPZW6ljPMfhQFkCGueF6x7wanfr f+BA== X-Gm-Message-State: AOAM533cwg+sitwI7YHvf5J50UEOlh72xDdGTaTX+Kr5gmN2qPyPdK2c +/Fxu4HUmPJHiwEjVjFNEv0XAzy6HWqNxg== X-Received: by 2002:a17:90b:2250:: with SMTP id hk16mr22832480pjb.95.1622916503546; Sat, 05 Jun 2021 11:08:23 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.157.221.215]) by smtp.googlemail.com with ESMTPSA id w14sm7818176pjf.12.2021.06.05.11.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 05 Jun 2021 11:08:23 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 5 Jun 2021 23:38:02 +0530 Message-Id: <20210605180806.65831-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 1/5] lavfi/dnn: Extract TaskItem and InferenceItem from OpenVino Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: AHo8zwlXqkzk Extract TaskItem and InferenceItem from OpenVino backend and convert ov_model to void in TaskItem. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 19 +++++++++ libavfilter/dnn/dnn_backend_openvino.c | 58 ++++++++++---------------- 2 files changed, 40 insertions(+), 37 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index cd9c0f5339..0c043e51f0 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -26,6 +26,25 @@ #include "../dnn_interface.h" +// one task for one function call from dnn interface +typedef struct TaskItem { + void *model; // model for the backend + AVFrame *in_frame; + AVFrame *out_frame; + const char *input_name; + const char *output_name; + int async; + int do_ioproc; + uint32_t inference_todo; + uint32_t inference_done; +} TaskItem; + +// one task might have multiple inferences +typedef struct InferenceItem { + TaskItem *task; + uint32_t bbox_index; +} InferenceItem; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 58c4ec9c9b..a84370d689 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -59,25 +59,6 @@ typedef struct OVModel{ Queue *inference_queue; // holds InferenceItem } OVModel; -// one task for one function call from dnn interface -typedef struct TaskItem { - OVModel *ov_model; - const char *input_name; - AVFrame *in_frame; - const char *output_name; - AVFrame *out_frame; - int do_ioproc; - int async; - uint32_t inference_todo; - uint32_t inference_done; -} TaskItem; - -// one task might have multiple inferences -typedef struct InferenceItem { - TaskItem *task; - uint32_t bbox_index; -} InferenceItem; - // one request for one call to openvino typedef struct RequestItem { ie_infer_request_t *infer_request; @@ -184,7 +165,7 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model, RequestItem *request request->inferences[i] = inference; request->inference_count = i + 1; task = inference->task; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { if (ov_model->model->frame_pre_proc != NULL) { @@ -220,11 +201,12 @@ static void infer_completion_callback(void *args) RequestItem *request = args; InferenceItem *inference = request->inferences[0]; TaskItem *task = inference->task; - SafeQueue *requestq = task->ov_model->request_queue; + OVModel *ov_model = task->model; + SafeQueue *requestq = ov_model->request_queue; ie_blob_t *output_blob = NULL; ie_blob_buffer_t blob_buffer; DNNData output; - OVContext *ctx = &task->ov_model->ctx; + OVContext *ctx = &ov_model->ctx; status = ie_infer_request_get_blob(request->infer_request, task->output_name, &output_blob); if (status != OK) { @@ -233,9 +215,9 @@ static void infer_completion_callback(void *args) char *all_output_names = NULL; size_t model_output_count = 0; av_log(ctx, AV_LOG_ERROR, "Failed to get model output data\n"); - status = ie_network_get_outputs_number(task->ov_model->network, &model_output_count); + status = ie_network_get_outputs_number(ov_model->network, &model_output_count); for (size_t i = 0; i < model_output_count; i++) { - status = ie_network_get_output_name(task->ov_model->network, i, &model_output_name); + status = ie_network_get_output_name(ov_model->network, i, &model_output_name); APPEND_STRING(all_output_names, model_output_name) } av_log(ctx, AV_LOG_ERROR, @@ -271,11 +253,11 @@ static void infer_completion_callback(void *args) task = request->inferences[i]->task; task->inference_done++; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (task->ov_model->model->frame_post_proc != NULL) { - task->ov_model->model->frame_post_proc(task->out_frame, &output, task->ov_model->model->filter_ctx); + if (ov_model->model->frame_post_proc != NULL) { + ov_model->model->frame_post_proc(task->out_frame, &output, ov_model->model->filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, &output, ctx); } @@ -285,18 +267,18 @@ static void infer_completion_callback(void *args) } break; case DFT_ANALYTICS_DETECT: - if (!task->ov_model->model->detect_post_proc) { + if (!ov_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); return; } - task->ov_model->model->detect_post_proc(task->out_frame, &output, 1, task->ov_model->model->filter_ctx); + ov_model->model->detect_post_proc(task->out_frame, &output, 1, ov_model->model->filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: - if (!task->ov_model->model->classify_post_proc) { + if (!ov_model->model->classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); return; } - task->ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, task->ov_model->model->filter_ctx); + ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, ov_model->model->filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -445,6 +427,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) InferenceItem *inference; TaskItem *task; OVContext *ctx; + OVModel *ov_model; if (ff_queue_size(inferenceq) == 0) { return DNN_SUCCESS; @@ -452,10 +435,11 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) inference = ff_queue_peek_front(inferenceq); task = inference->task; - ctx = &task->ov_model->ctx; + ov_model = task->model; + ctx = &ov_model->ctx; if (task->async) { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -471,7 +455,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) } return DNN_SUCCESS; } else { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -694,7 +678,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu task.in_frame = in_frame; task.output_name = output_name; task.out_frame = out_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { av_frame_free(&out_frame); @@ -814,7 +798,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * task.in_frame = exec_params->in_frame; task.output_name = exec_params->output_names[0]; task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, exec_params) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); @@ -861,7 +845,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa task->in_frame = exec_params->in_frame; task->output_name = exec_params->output_names[0]; task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task->ov_model = ov_model; + task->model = ov_model; if (ff_queue_push_back(ov_model->task_queue, task) < 0) { av_freep(&task); av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n");