From patchwork Fri May 28 09:24:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27971 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp384806iof; Fri, 28 May 2021 02:55:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzg8mjTfMB1OTkuC3pEwfmh71pBs2ace+CqgNjJKMr4bDhYA3zmDyu6jt+leh7lf4a5KepZ X-Received: by 2002:a05:6402:19a:: with SMTP id r26mr8897940edv.44.1622195715211; Fri, 28 May 2021 02:55:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622195715; cv=none; d=google.com; s=arc-20160816; b=V3JSVb8SV/5EOzl2gShj+Fn8EAJ6sXWlgcCp2LUx9sn92z0WpRg5GsUYsn20Kqa1hn Vw09hqHtg7RuTN4asihpC/f+CHcx23qTuph3jLAJKioL0CS+W1Esfk/EitQI9dYz+i5Y xhPLDraLz0A+PPQxM6c8uN2N/sGeJdNJr51VIPR54YkC5+svWgBOk0l8C59cVtPGmWTz MERsz8hqNkh5Vd2KTafh2IZlEV42QIJNRgALzARs8klXI/woLmLKz7jPsmsbR0z/7GYW gCw3b/QVWLb2OI1qjyGB5Ew/saZ7nKGDGWIpVA8zU9nTX8uKThTJWd1PsD9KSdtxjBSd 8vZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=GEo/0/1XrAdrKdht1l/eg+5SqgjoYss5ufrRkgIYRus=; b=cTG6/pK2Igx7R/FokHMstel2lK4tvm6cdmq135OhHIwez7yxbubvIqhiYli88JLRLv eVN5JOR24kKItlZPa2phPYg8q84MhrBus8GwMyWdOaQpx6biDMi4RvqlLM2Ho/T/cwW+ eZyHaGH5ltJJ1Kpl0qguZ+aRCcXu3ujYDxmgHLXUCYN7wnYxlnoUlfyzvg/RhwmJf6T4 SZHiSS4X34pigaqSwI7NBlAaeSn1cNe5eQFd4dDMxRi/sT4yE+xXeWnqqxaifRt381qG u/LnDBWXnSxkrYCwFHSUk3UhDbVlJqE60hfSEanPqdneSQz+K2s4CYkok4KzZ6hyZLAV srCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="TKXTs/wg"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fi2si4635362ejb.14.2021.05.28.02.55.14; Fri, 28 May 2021 02:55:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="TKXTs/wg"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1D56168A26A; Fri, 28 May 2021 12:55:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9DE6B680911 for ; Fri, 28 May 2021 12:55:05 +0300 (EEST) Received: by mail-pg1-f169.google.com with SMTP id i5so2182493pgm.0 for ; Fri, 28 May 2021 02:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=TKXTs/wg0W14U+kO3ETpZKVF4FnkIcIUufYyQevK1dQxMQjwH+k3AGdSnp2GCOWNju 99Bhs6P1slEPCX0LpOw1G+f5Q2AZnI5ZwQ1XJuYdpJVYasAn+y8IMYB3yz8l+DNLTh/G Q0cfPTZnZd+N6GcauqYg9z7ZgA9cs1XEXXNzx3uSHYUsrqXvxLRZP+wWEqs5710RJFDS qYqBj6LZTwxVJNRl/9zQb5DnqHpO3zoXwhqOTLwP7hN3vto1dwvgOXwZqPnyzeUBHzul B3maEOnekBpbuaxu8pIb05uz1sOtMX1kyz4OsQ+I/hdUBVTUR6Mfg8yoi3UOkZHEYno3 BPgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=WVvMPEwGlWhYKc75RCUcE2876Rr8ZSO3qzVTpcXE9R95R3pK4/UDkmFh0e3TOZzweC ghu3G5j15WHv8Vh+K7BXgFPFq9jBuidBCmdWJCHhwg2ibw8hSm2ViACuSqupaDAtpCpI 33XzsFlLVn/cQZOixIVEwCoQ+e70KcTcEaF5IGvCBcAAbza7Mvv7w0XQAB/s0u9JfJ0F WVvC3WzE4eYx5lNiQKwp503kyMKckTPgZpSdQeALOf/h2V4FH/NSW6fwve61CtG7pi04 YHpgoNfqu+KziL6acvp34MsegUofwUECM2wRJXccnbvqW9pkAUqD9PuwO8TByOHDXV0+ 5TNA== X-Gm-Message-State: AOAM530rSnziRcgaQ0uTbaCJ6mwjXsgYz3Pb5dv5Qlr1q/8zqmXy17TT RvWVrzX+/jFz8aCKkRsazsK+7Wgr91+ZyQ== X-Received: by 2002:a17:902:7887:b029:ef:9429:e956 with SMTP id q7-20020a1709027887b02900ef9429e956mr7308325pll.78.1622193947946; Fri, 28 May 2021 02:25:47 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:47 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:45 +0530 Message-Id: <20210528092454.31874-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 01/10] lavfi/dnn: Extract TaskItem and InferenceItem from OpenVino Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 7OTlDQLsG0rq Extract TaskItem and InferenceItem from OpenVino backend and convert ov_model to void in TaskItem. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 19 +++++++++ libavfilter/dnn/dnn_backend_openvino.c | 58 ++++++++++---------------- 2 files changed, 40 insertions(+), 37 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index cd9c0f5339..0c043e51f0 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -26,6 +26,25 @@ #include "../dnn_interface.h" +// one task for one function call from dnn interface +typedef struct TaskItem { + void *model; // model for the backend + AVFrame *in_frame; + AVFrame *out_frame; + const char *input_name; + const char *output_name; + int async; + int do_ioproc; + uint32_t inference_todo; + uint32_t inference_done; +} TaskItem; + +// one task might have multiple inferences +typedef struct InferenceItem { + TaskItem *task; + uint32_t bbox_index; +} InferenceItem; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 58c4ec9c9b..a84370d689 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -59,25 +59,6 @@ typedef struct OVModel{ Queue *inference_queue; // holds InferenceItem } OVModel; -// one task for one function call from dnn interface -typedef struct TaskItem { - OVModel *ov_model; - const char *input_name; - AVFrame *in_frame; - const char *output_name; - AVFrame *out_frame; - int do_ioproc; - int async; - uint32_t inference_todo; - uint32_t inference_done; -} TaskItem; - -// one task might have multiple inferences -typedef struct InferenceItem { - TaskItem *task; - uint32_t bbox_index; -} InferenceItem; - // one request for one call to openvino typedef struct RequestItem { ie_infer_request_t *infer_request; @@ -184,7 +165,7 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model, RequestItem *request request->inferences[i] = inference; request->inference_count = i + 1; task = inference->task; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { if (ov_model->model->frame_pre_proc != NULL) { @@ -220,11 +201,12 @@ static void infer_completion_callback(void *args) RequestItem *request = args; InferenceItem *inference = request->inferences[0]; TaskItem *task = inference->task; - SafeQueue *requestq = task->ov_model->request_queue; + OVModel *ov_model = task->model; + SafeQueue *requestq = ov_model->request_queue; ie_blob_t *output_blob = NULL; ie_blob_buffer_t blob_buffer; DNNData output; - OVContext *ctx = &task->ov_model->ctx; + OVContext *ctx = &ov_model->ctx; status = ie_infer_request_get_blob(request->infer_request, task->output_name, &output_blob); if (status != OK) { @@ -233,9 +215,9 @@ static void infer_completion_callback(void *args) char *all_output_names = NULL; size_t model_output_count = 0; av_log(ctx, AV_LOG_ERROR, "Failed to get model output data\n"); - status = ie_network_get_outputs_number(task->ov_model->network, &model_output_count); + status = ie_network_get_outputs_number(ov_model->network, &model_output_count); for (size_t i = 0; i < model_output_count; i++) { - status = ie_network_get_output_name(task->ov_model->network, i, &model_output_name); + status = ie_network_get_output_name(ov_model->network, i, &model_output_name); APPEND_STRING(all_output_names, model_output_name) } av_log(ctx, AV_LOG_ERROR, @@ -271,11 +253,11 @@ static void infer_completion_callback(void *args) task = request->inferences[i]->task; task->inference_done++; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (task->ov_model->model->frame_post_proc != NULL) { - task->ov_model->model->frame_post_proc(task->out_frame, &output, task->ov_model->model->filter_ctx); + if (ov_model->model->frame_post_proc != NULL) { + ov_model->model->frame_post_proc(task->out_frame, &output, ov_model->model->filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, &output, ctx); } @@ -285,18 +267,18 @@ static void infer_completion_callback(void *args) } break; case DFT_ANALYTICS_DETECT: - if (!task->ov_model->model->detect_post_proc) { + if (!ov_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); return; } - task->ov_model->model->detect_post_proc(task->out_frame, &output, 1, task->ov_model->model->filter_ctx); + ov_model->model->detect_post_proc(task->out_frame, &output, 1, ov_model->model->filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: - if (!task->ov_model->model->classify_post_proc) { + if (!ov_model->model->classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); return; } - task->ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, task->ov_model->model->filter_ctx); + ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, ov_model->model->filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -445,6 +427,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) InferenceItem *inference; TaskItem *task; OVContext *ctx; + OVModel *ov_model; if (ff_queue_size(inferenceq) == 0) { return DNN_SUCCESS; @@ -452,10 +435,11 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) inference = ff_queue_peek_front(inferenceq); task = inference->task; - ctx = &task->ov_model->ctx; + ov_model = task->model; + ctx = &ov_model->ctx; if (task->async) { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -471,7 +455,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) } return DNN_SUCCESS; } else { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -694,7 +678,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu task.in_frame = in_frame; task.output_name = output_name; task.out_frame = out_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { av_frame_free(&out_frame); @@ -814,7 +798,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * task.in_frame = exec_params->in_frame; task.output_name = exec_params->output_names[0]; task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, exec_params) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); @@ -861,7 +845,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa task->in_frame = exec_params->in_frame; task->output_name = exec_params->output_names[0]; task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task->ov_model = ov_model; + task->model = ov_model; if (ff_queue_push_back(ov_model->task_queue, task) < 0) { av_freep(&task); av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n");