From patchwork Mon May 17 05:54:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Yejun" X-Patchwork-Id: 27819 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp2550307iof; Sun, 16 May 2021 23:07:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzMiiNDznbuV16zztUcHrtfa6HhNHNavhNi46Ci/mnDMiZQq8bhQTj4zXvWi00fzXWQ+Xl4 X-Received: by 2002:a05:6402:134d:: with SMTP id y13mr8127631edw.287.1621231644376; Sun, 16 May 2021 23:07:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1621231644; cv=none; d=google.com; s=arc-20160816; b=ZG1PDLyBb+ceXD9URztDCFRiwrGvoLJn/YRMvCEe1AVKz5by5oPkrD8adsb/rwauY0 LcdDncboOfhHqwmKFF5+hOySvI/1AkLcDCY3cMIB4GmHmp5xPbviPv7kJBUnH6Nm/H59 7X6TD82Efxl7+gTFnxppsJeFv4NzpEJ+zb1+o3gDSNL5Zsm4O6E8R+OxIHR9mxYrPgXz xAtD8wUcGvUsMPcAj8Svwo95HVfXjKGkhS7NlvDxRMJ+uWGJUk9+ox3eyI691WCv+O7i MLYDQJEi5z8n9ySX+Tuz7IQcIHPHzifyk0wc0kqSLmn0riBhYSkT1dIjCsHtJ65g/I4S SpQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=XcrVB37S8v3jdKeY6fDGdJXKCGIa8hrYArMqZuwe1IM=; b=LFNMEBP/9z4P9bEp6lo4Vd3wvmfBht4UIdQsNyE2VEVivcAmstN1DInjsyF7bkZvL+ RWtOeHvjhYqeu2vNHMKxRm6pbXow68ycBdZteDitsV3VUKBocOxBBJkoGD/YJXW96XWL j7MQWPFKVH7bN6b3AjbKmmwIeeRwW62r2h9bxcEoiiMAnY7PmTkChGAih6XKeaz2i2mt WZDAAz+fze6t4nHznW/FkL+4gYqGwG4P5gKKlbfdyngpve2QGxYezG+QcZ6oCqJlH5qg 393f97HSRUM1Z/Dlcu/T2iR6yKNHR7pdYCG4Pc0+jPXamo0/S0kleiK5DesyPGbbPM12 EFXA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y13si13691502edp.239.2021.05.16.23.07.24; Sun, 16 May 2021 23:07:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E42726883C9; Mon, 17 May 2021 09:07:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2B9B4689806 for ; Mon, 17 May 2021 09:07:07 +0300 (EEST) IronPort-SDR: nNXChwpmBNr52yDsWK3kdxdA7bPkmIkz/VNYOxlSp+TNbgH7wuI7fircDK0WQBpWXgcwLagsmy QYSqSL9N2Q6g== X-IronPort-AV: E=McAfee;i="6200,9189,9986"; a="200442486" X-IronPort-AV: E=Sophos;i="5.82,306,1613462400"; d="scan'208";a="200442486" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 May 2021 23:06:56 -0700 IronPort-SDR: HqBbnYJub01YrDdIV+bDSKD0N5wUqs3lqmMi03EsoHVAo0itXMcitPgMoj0WbK1xvwS9OxRDqX QnLQE81l1DWw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,306,1613462400"; d="scan'208";a="540265335" Received: from yguo18-skl-u1604.sh.intel.com ([10.239.159.53]) by fmsmga001.fm.intel.com with ESMTP; 16 May 2021 23:06:55 -0700 From: "Guo, Yejun" To: ffmpeg-devel@ffmpeg.org Date: Mon, 17 May 2021 13:54:11 +0800 Message-Id: <20210517055411.8654-2-yejun.guo@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210517055411.8654-1-yejun.guo@intel.com> References: <20210517055411.8654-1-yejun.guo@intel.com> Subject: [FFmpeg-devel] [PATCH 2/2] lavfi/dnn: refine code to separate processing and detection in backends X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: yejun.guo@intel.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: AcJxR0y/pMme --- libavfilter/dnn/dnn_backend_native.c | 2 +- libavfilter/dnn/dnn_backend_openvino.c | 6 ++++-- libavfilter/dnn/dnn_backend_tf.c | 20 +++++++++++++++----- libavfilter/dnn/dnn_io_proc.c | 18 ++---------------- libavfilter/dnn/dnn_io_proc.h | 3 ++- 5 files changed, 24 insertions(+), 25 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native.c b/libavfilter/dnn/dnn_backend_native.c index b5f1c16538..a6be27f1fd 100644 --- a/libavfilter/dnn/dnn_backend_native.c +++ b/libavfilter/dnn/dnn_backend_native.c @@ -314,7 +314,7 @@ static DNNReturnType execute_model_native(const DNNModel *model, const char *inp if (native_model->model->frame_pre_proc != NULL) { native_model->model->frame_pre_proc(in_frame, &input, native_model->model->filter_ctx); } else { - ff_proc_from_frame_to_dnn(in_frame, &input, native_model->model->func_type, ctx); + ff_proc_from_frame_to_dnn(in_frame, &input, ctx); } } diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 1ff8a720b9..e0781e854a 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -186,15 +186,17 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model, RequestItem *request task = inference->task; switch (task->ov_model->model->func_type) { case DFT_PROCESS_FRAME: - case DFT_ANALYTICS_DETECT: if (task->do_ioproc) { if (ov_model->model->frame_pre_proc != NULL) { ov_model->model->frame_pre_proc(task->in_frame, &input, ov_model->model->filter_ctx); } else { - ff_proc_from_frame_to_dnn(task->in_frame, &input, ov_model->model->func_type, ctx); + ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } } break; + case DFT_ANALYTICS_DETECT: + ff_frame_to_dnn_detect(task->in_frame, &input, ctx); + break; case DFT_ANALYTICS_CLASSIFY: ff_frame_to_dnn_classify(task->in_frame, &input, inference->bbox_index, ctx); break; diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 5908aeb359..4c16c2bdb0 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -763,12 +763,22 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n } input.data = (float *)TF_TensorData(input_tensor); - if (do_ioproc) { - if (tf_model->model->frame_pre_proc != NULL) { - tf_model->model->frame_pre_proc(in_frame, &input, tf_model->model->filter_ctx); - } else { - ff_proc_from_frame_to_dnn(in_frame, &input, tf_model->model->func_type, ctx); + switch (tf_model->model->func_type) { + case DFT_PROCESS_FRAME: + if (do_ioproc) { + if (tf_model->model->frame_pre_proc != NULL) { + tf_model->model->frame_pre_proc(in_frame, &input, tf_model->model->filter_ctx); + } else { + ff_proc_from_frame_to_dnn(in_frame, &input, ctx); + } } + break; + case DFT_ANALYTICS_DETECT: + ff_frame_to_dnn_detect(in_frame, &input, ctx); + break; + default: + avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model->func_type); + break; } tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs)); diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 1e2bef3f9a..e01661103b 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -94,7 +94,7 @@ DNNReturnType ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *l return DNN_SUCCESS; } -static DNNReturnType proc_from_frame_to_dnn_frameprocessing(AVFrame *frame, DNNData *input, void *log_ctx) +DNNReturnType ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, void *log_ctx) { struct SwsContext *sws_ctx; int bytewidth = av_image_get_linesize(frame->format, frame->width, 0); @@ -243,7 +243,7 @@ DNNReturnType ff_frame_to_dnn_classify(AVFrame *frame, DNNData *input, uint32_t return DNN_SUCCESS; } -static DNNReturnType proc_from_frame_to_dnn_analytics(AVFrame *frame, DNNData *input, void *log_ctx) +DNNReturnType ff_frame_to_dnn_detect(AVFrame *frame, DNNData *input, void *log_ctx) { struct SwsContext *sws_ctx; int linesizes[4]; @@ -271,17 +271,3 @@ static DNNReturnType proc_from_frame_to_dnn_analytics(AVFrame *frame, DNNData *i sws_freeContext(sws_ctx); return DNN_SUCCESS; } - -DNNReturnType ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, DNNFunctionType func_type, void *log_ctx) -{ - switch (func_type) - { - case DFT_PROCESS_FRAME: - return proc_from_frame_to_dnn_frameprocessing(frame, input, log_ctx); - case DFT_ANALYTICS_DETECT: - return proc_from_frame_to_dnn_analytics(frame, input, log_ctx); - default: - avpriv_report_missing_feature(log_ctx, "model function type %d", func_type); - return DNN_ERROR; - } -} diff --git a/libavfilter/dnn/dnn_io_proc.h b/libavfilter/dnn/dnn_io_proc.h index 16dcdd6d1a..daef01aceb 100644 --- a/libavfilter/dnn/dnn_io_proc.h +++ b/libavfilter/dnn/dnn_io_proc.h @@ -30,8 +30,9 @@ #include "../dnn_interface.h" #include "libavutil/frame.h" -DNNReturnType ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, DNNFunctionType func_type, void *log_ctx); +DNNReturnType ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, void *log_ctx); DNNReturnType ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx); +DNNReturnType ff_frame_to_dnn_detect(AVFrame *frame, DNNData *input, void *log_ctx); DNNReturnType ff_frame_to_dnn_classify(AVFrame *frame, DNNData *input, uint32_t bbox_index, void *log_ctx); #endif