From patchwork Sat Apr 27 16:41:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48298 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1069915pzb; Sat, 27 Apr 2024 09:42:46 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXgOggNJTgaYhMuQYk5kgSxoA4a3UjlMKgnvitqW7Jm4Db8vBhlJI6EmjqK1kgbq5Ixe+YuBKTDbncpQGmOOrqJp6LYW4O/WPEA/Q== X-Google-Smtp-Source: AGHT+IF5eTLDTJxIjNKRag8lE6483umnf2f90hAB5bZUu4vtPrrUEC1ZMuBoykey2UPvqLf4176z X-Received: by 2002:ac2:59c1:0:b0:51a:f2fb:b13c with SMTP id x1-20020ac259c1000000b0051af2fbb13cmr3184887lfn.11.1714236166023; Sat, 27 Apr 2024 09:42:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236165; cv=none; d=google.com; s=arc-20160816; b=dB/1umdfjPHAQJB7bQyVFJK5Nj7O6opc9RMRLXGwXhrAQmzg3c/k+okP2Rg9me9tob 7bYb5HT1b/50L1usqP2+SWTgZq2dodWStpF4tomFqcblVf2F3ec5PHDynl5EnITHE+ob wOY8xufOIsdX//U/njNDTvoB4rRAtrv+GMNXwWNXPx8/T3SbvKQY9zxpiDs4tE2qCwXS 4/IyaFBXSQLjgbTxw47QXytPEVDO7hnAMYWIoMgN21xDWiV+iApieuMPW7AWsiHeknDu 2du9skVH/ugt7YJRsn7CNi2lm7We1ll38hmj+cPV0ZRGNr14JG1nBA5kAkEnHlhVAHR5 z1RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=GLO40fPCrG0+EOZ1tC8C6ocmMFLODRitfMuba8zQylg=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=B8U1kMOg3xUwiBHPVmR8A1o8TvsDhTWRFzaRyGav+Szm5DxjAcmh2btzMHdhN804E8 YEjAx5XGdtUEEMYt4VQKvDD/AcNzTgjFojqNYrhsJUoSGe4xwKvQTAQApGCRmE78jDVt RctqRhGZfCMqny2TLCYd0FCefNOm8cl6d5Ujt83IC5OCTzYG4dQs9dTqV/J+Ghpbj+M1 2+qRFSl2hfP/yciqxAiIE/LVOfVid2pk95qqdTkuhpGRINw2Azr3OA3Lb1E5dDPNpcgI aERafJ61ERVPGBMNpVRZRQeGcgaAmgXWwJa7h/ABFuI/8B9mlCCJnEdrMJbnuEqh8B58 bgkg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="VpUqhjW/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l13-20020aa7d94d000000b005726b19bc50si1027768eds.47.2024.04.27.09.42.45; Sat, 27 Apr 2024 09:42:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="VpUqhjW/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C0A8D68D377; Sat, 27 Apr 2024 19:42:23 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B3E2668D294 for ; Sat, 27 Apr 2024 19:42:13 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236123; bh=g0CdU38uBJ2uE8UGL1myBq6QQ2LYD3zexywOMRMApO8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=VpUqhjW/Gs5BFOfFFwKpIgLPhI4JI548QgXw03fFwg4i2mi6IFyy5b/tf84R7JgrR l7CZhWu2DE4MRyswDF5r0T5Ds0AKWAtfcOdNs4zcYLqz1jqgDJO2lc8Z2t5J5VQ4hZ 8cB1HN+WkBXk14krZVGNvj41FDjcczMLVzhmBOc4= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236122ty8kz81if Message-ID: X-QQ-XMAILINFO: MFGUwB2gEJ8h3MhzNIMPHLJfxn8VSs1j/mN9nZJne0t79v0PpWzly/RsDd47Ra oVf2bOtWoJYhjtyi1HsH5VcGEkUwSd/uqU4eOYHDZNBKmXen8JhS8ciH8DizKpMVP09yMr+PgCp7 F/bNObQLSsBXadt8yiFrQVwz4/6NO+Cx39AufAtKpFTN2+V5VgW2pNDzRnqSuoU0LmnRlEbSwp4H 0C0dC/HMM6fYr0wKM06IQyfRmvXe538w3sApdbfHkVY43Bt+7uwy3aGl0C2l5/VUXjJdzeaYiNy/ qBtGeB00b7/LzoFn8R+aLZajkRw10J4MG5Ux5Hdu7F00Ap68IZbJZ7Sl6kgl/3+zizco/qV9Tc6q 8cIwpsIKRtgrhoOCkVSIjFl3vrpRpfsJL+W8EaSJZwGH778xG6MxMx2tVPsCZ5YoJSLkcyLtYHDF cKIFTiVHXuUicnymVtMFkkumUTm28kDqlZNP5OxxyFVhVoG7BYW5OfG0GZE9pPCBv9n8pjVnYryC UcvbAUxQnLHlYPMukanPXaYhk7Zrcysr6pb1JWsWCv1K4Tc/65TtWPJeN8Lnl7cFUVXJIrX3cmvs K5HPuDGJObyVjISqNN7qW7GwPqhC8ghwNOjvrjgfEhTc33j+wnLX3KotKm0bOeGhcByRKPkzdsRZ 3rB26kx2VGh3iJ3dNhof2/fcJglKQs12Xwy0iHmEdpLsFtC5ax93uy5VwRoaEyL/A+OT5Ft1aeup mcnGfUZt9FDKacW9ETMSUudyzaudYxOtwu+4lPAPj62oRbMjnRhxJv5MUOp9Ih/a39/bxycTfPlJ r1ttp2j3NqQD+pWPjzQDQ0JYv1utnZYdJXfJSUWRDUyXNJIQyD/PMhga9hKor0DItTCd2iXKv87t SJwdenu3xsyTdOKbHjW+vxHo4VVXLKLxHWAXl6GjhpcAtH8arbHblWUFVaOryG2KNOoJ4pFRYw8e 36ZacaqeatXLDtLL92tKDNFdT0ZpdixA1KTJLvP+3IpQpCJigPF1w1g3v4SHBlt/cDVRBjzwA= X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:51 +0800 X-OQ-MSGID: <20240427164159.82771-2-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 1/9] avfilter/dnn: Refactor DNN parameter configuration system X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: tbhK5rMaMvKS From: Zhao Zhili This patch trying to resolve mulitiple issues related to parameter configuration: Firstly, each DNN filters duplicate DNN_COMMON_OPTIONS, which should be the common options of backend. Secondly, backend options are hidden behind the scene. It's a AV_OPT_TYPE_STRING backend_configs for user, and parsed by each backend. We don't know each backend support what kind of options from the help message. Third, DNN backends duplicate DNN_BACKEND_COMMON_OPTIONS. Last but not the least, pass backend options via AV_OPT_TYPE_STRING makes it hard to pass AV_OPT_TYPE_BINARY to backend, if not impossible. This patch puts backend common options and each backend options inside DnnContext to reduce code duplication, make options user friendly, and easy to extend for future usecase. There is a known issue that, for a filter which only support one or two of the backends, the help message still show the option of all three backend. Each DNN filter should be able to run on any backend. Current issue is mostly due to incomplete implementation of backend (e.g., only support DFT_PROCESS_FRAME with libtorch backedn), and lack of maintenance on the filters. For example, ./ffmpeg -h filter=dnn_processing dnn_processing AVOptions: dnn_backend ..FV....... DNN backend (from INT_MIN to INT_MAX) (default tensorflow) tensorflow 1 ..FV....... tensorflow backend flag openvino 2 ..FV....... openvino backend flag torch 3 ..FV....... torch backend flag dnn_base AVOptions: model ..F........ path to model file input ..F........ input name of the model output ..F........ output name of the model backend_configs ..F.......P backend configs (deprecated) options ..F.......P backend configs (deprecated) async ..F........ use DNN async inference (ignored, use backend_configs='async=1') (default true) nireq ..F........ number of request (from 0 to INT_MAX) (default 0) async ..F........ use DNN async inference (default true) device ..F........ device to run model dnn_tensorflow AVOptions: sess_config ..F........ config for SessionOptions dnn_openvino AVOptions: batch_size ..F........ batch size per request (from 1 to 1000) (default 1) input_resizable ..F........ can input be resizable or not (default false) layout ..F........ input layout of model (from 0 to 2) (default none) none 0 ..F........ none nchw 1 ..F........ nchw nhwc 2 ..F........ nhwc scale ..F........ Add scale preprocess operation. Divide each element of input by specified value. (from INT_MIN to INT_MAX) (default 0) mean ..F........ Add mean preprocess operation. Subtract specified value from each element of input. (from INT_MIN to INT_MAX) (default 0) dnn_th AVOptions: optimize ..F........ turn on graph executor optimization (from 0 to 1) (default 0) --- libavfilter/dnn/dnn_backend_common.h | 13 ++- libavfilter/dnn/dnn_backend_openvino.c | 146 ++++++++++--------------- libavfilter/dnn/dnn_backend_tf.c | 82 +++++--------- libavfilter/dnn/dnn_backend_torch.cpp | 67 ++++-------- libavfilter/dnn/dnn_interface.c | 81 ++++++++++++++ libavfilter/dnn_filter_common.c | 32 +++++- libavfilter/dnn_filter_common.h | 37 +++---- libavfilter/dnn_interface.h | 66 ++++++++++- libavfilter/vf_derain.c | 5 +- libavfilter/vf_dnn_classify.c | 3 +- libavfilter/vf_dnn_detect.c | 3 +- libavfilter/vf_dnn_processing.c | 3 +- libavfilter/vf_sr.c | 5 +- 13 files changed, 314 insertions(+), 229 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 42c67c7040..9f5d37b3e0 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -28,9 +28,16 @@ #include "../dnn_interface.h" #include "libavutil/thread.h" -#define DNN_BACKEND_COMMON_OPTIONS \ - { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, \ - { "async", "use DNN async inference", OFFSET(options.async), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, +#define DNN_DEFINE_CLASS_EXT(name, desc, options) \ + { \ + .class_name = desc, \ + .item_name = av_default_item_name, \ + .option = options, \ + .version = LIBAVUTIL_VERSION_INT, \ + .category = AV_CLASS_CATEGORY_FILTER, \ + } +#define DNN_DEFINE_CLASS(fname) \ + DNN_DEFINE_CLASS_EXT(fname, #fname, fname##_options) // one task for one function call from dnn interface typedef struct TaskItem { diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 374f21b7a1..c4b0682f11 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -40,24 +40,8 @@ #endif #include "dnn_backend_common.h" -typedef struct OVOptions{ - char *device_type; - int nireq; - uint8_t async; - int batch_size; - int input_resizable; - DNNLayout layout; - float scale; - float mean; -} OVOptions; - -typedef struct OVContext { - const AVClass *class; - OVOptions options; -} OVContext; - typedef struct OVModel{ - OVContext ctx; + DnnContext *ctx; DNNModel *model; #if HAVE_OPENVINO2 ov_core_t *core; @@ -98,24 +82,20 @@ typedef struct OVRequestItem { generated_string = generated_string ? av_asprintf("%s %s", generated_string, iterate_string) : \ av_asprintf("%s", iterate_string); -#define OFFSET(x) offsetof(OVContext, x) +#define OFFSET(x) offsetof(OVOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_openvino_options[] = { - { "device", "device to run model", OFFSET(options.device_type), AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS }, - DNN_BACKEND_COMMON_OPTIONS - { "batch_size", "batch size per request", OFFSET(options.batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, - { "input_resizable", "can input be resizable or not", OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, - { "layout", "input layout of model", OFFSET(options.layout), AV_OPT_TYPE_INT, { .i64 = DL_NONE}, DL_NONE, DL_NHWC, FLAGS, .unit = "layout" }, + { "batch_size", "batch size per request", OFFSET(batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, + { "input_resizable", "can input be resizable or not", OFFSET(input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "layout", "input layout of model", OFFSET(layout), AV_OPT_TYPE_INT, { .i64 = DL_NONE}, DL_NONE, DL_NHWC, FLAGS, .unit = "layout" }, { "none", "none", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NONE }, 0, 0, FLAGS, .unit = "layout"}, { "nchw", "nchw", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NCHW }, 0, 0, FLAGS, .unit = "layout"}, { "nhwc", "nhwc", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NHWC }, 0, 0, FLAGS, .unit = "layout"}, - { "scale", "Add scale preprocess operation. Divide each element of input by specified value.", OFFSET(options.scale), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, - { "mean", "Add mean preprocess operation. Subtract specified value from each element of input.", OFFSET(options.mean), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, + { "scale", "Add scale preprocess operation. Divide each element of input by specified value.", OFFSET(scale), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, + { "mean", "Add mean preprocess operation. Subtract specified value from each element of input.", OFFSET(mean), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_openvino); - #if HAVE_OPENVINO2 static const struct { ov_status_e status; @@ -199,7 +179,7 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) DNNData input; LastLevelTaskItem *lltask; TaskItem *task; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 int64_t* dims; ov_status_e status; @@ -292,7 +272,7 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) input.scale = 1; input.mean = 0; - for (int i = 0; i < ctx->options.batch_size; ++i) { + for (int i = 0; i < ctx->ov_option.batch_size; ++i) { lltask = ff_queue_pop_front(ov_model->lltask_queue); if (!lltask) { break; @@ -360,7 +340,7 @@ static void infer_completion_callback(void *args) OVModel *ov_model = task->model; SafeQueue *requestq = ov_model->request_queue; DNNData *outputs; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 size_t* dims; ov_status_e status; @@ -410,9 +390,9 @@ static void infer_completion_callback(void *args) outputs[i].dims[2] = output_shape.rank > 1 ? dims[output_shape.rank - 2] : 1; outputs[i].dims[3] = output_shape.rank > 0 ? dims[output_shape.rank - 1] : 1; av_assert0(request->lltask_count <= dims[0]); - outputs[i].layout = ctx->options.layout; - outputs[i].scale = ctx->options.scale; - outputs[i].mean = ctx->options.mean; + outputs[i].layout = ctx->ov_option.layout; + outputs[i].scale = ctx->ov_option.scale; + outputs[i].mean = ctx->ov_option.mean; ov_shape_free(&output_shape); ov_tensor_free(output_tensor); output_tensor = NULL; @@ -452,9 +432,9 @@ static void infer_completion_callback(void *args) output.dims[i] = dims.dims[i]; av_assert0(request->lltask_count <= dims.dims[0]); output.dt = precision_to_datatype(precision); - output.layout = ctx->options.layout; - output.scale = ctx->options.scale; - output.mean = ctx->options.mean; + output.layout = ctx->ov_option.layout; + output.scale = ctx->ov_option.scale; + output.mean = ctx->ov_option.mean; outputs = &output; #endif @@ -590,7 +570,6 @@ static void dnn_free_model_ov(DNNModel **model) av_free(ov_model->all_output_names); av_free(ov_model->all_input_names); #endif - av_opt_free(&ov_model->ctx); av_freep(&ov_model); av_freep(model); } @@ -599,7 +578,7 @@ static void dnn_free_model_ov(DNNModel **model) static int init_model_ov(OVModel *ov_model, const char *input_name, const char **output_names, int nb_outputs) { int ret = 0; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 ov_status_e status; ov_preprocess_input_tensor_info_t* input_tensor_info = NULL; @@ -610,7 +589,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_layout_t* NCHW_layout = NULL; const char* NHWC_desc = "NHWC"; const char* NCHW_desc = "NCHW"; - const char* device = ctx->options.device_type; + const char* device = ctx->device ? ctx->device : "CPU"; #else IEStatusCode status; ie_available_devices_t a_dev; @@ -618,17 +597,17 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * char *all_dev_names = NULL; #endif // We scale pixel by default when do frame processing. - if (fabsf(ctx->options.scale) < 1e-6f) - ctx->options.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; + if (fabsf(ctx->ov_option.scale) < 1e-6f) + ctx->ov_option.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; // batch size - if (ctx->options.batch_size <= 0) { - ctx->options.batch_size = 1; + if (ctx->ov_option.batch_size <= 0) { + ctx->ov_option.batch_size = 1; } #if HAVE_OPENVINO2 - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { avpriv_report_missing_feature(ctx, "Do not support batch_size > 1 for now," "change batch_size to 1.\n"); - ctx->options.batch_size = 1; + ctx->ov_option.batch_size = 1; } status = ov_preprocess_prepostprocessor_create(ov_model->ov_model, &ov_model->preprocess); @@ -677,9 +656,9 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ret = ov2_map_error(status, NULL); goto err; } - if (ctx->options.layout == DL_NCHW) + if (ctx->ov_option.layout == DL_NCHW) status = ov_preprocess_input_model_info_set_layout(input_model_info, NCHW_layout); - else if (ctx->options.layout == DL_NHWC) + else if (ctx->ov_option.layout == DL_NHWC) status = ov_preprocess_input_model_info_set_layout(input_model_info, NHWC_layout); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to get set input model layout\n"); @@ -725,7 +704,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } if (ov_model->model->func_type != DFT_PROCESS_FRAME) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); - else if (fabsf(ctx->options.scale - 1) > 1e-6f || fabsf(ctx->options.mean) > 1e-6f) + else if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); else status |= ov_preprocess_output_set_element_type(output_tensor_info, U8); @@ -740,7 +719,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_model->output_info = NULL; } // set preprocess steps. - if (fabsf(ctx->options.scale - 1) > 1e-6f || fabsf(ctx->options.mean) > 1e-6f) { + if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) { ov_preprocess_preprocess_steps_t* input_process_steps = NULL; status = ov_preprocess_input_info_get_preprocess_steps(ov_model->input_info, &input_process_steps); if (status != OK) { @@ -749,8 +728,8 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } status = ov_preprocess_preprocess_steps_convert_element_type(input_process_steps, F32); - status |= ov_preprocess_preprocess_steps_mean(input_process_steps, ctx->options.mean); - status |= ov_preprocess_preprocess_steps_scale(input_process_steps, ctx->options.scale); + status |= ov_preprocess_preprocess_steps_mean(input_process_steps, ctx->ov_option.mean); + status |= ov_preprocess_preprocess_steps_scale(input_process_steps, ctx->ov_option.scale); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to set preprocess steps\n"); ov_preprocess_preprocess_steps_free(input_process_steps); @@ -824,7 +803,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_layout_free(NCHW_layout); ov_layout_free(NHWC_layout); #else - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { input_shapes_t input_shapes; status = ie_network_get_input_shapes(ov_model->network, &input_shapes); if (status != OK) { @@ -832,7 +811,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } for (int i = 0; i < input_shapes.shape_num; i++) - input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size; + input_shapes.shapes[i].shape.dims[0] = ctx->ov_option.batch_size; status = ie_network_reshape(ov_model->network, input_shapes); ie_network_input_shapes_free(&input_shapes); if (status != OK) { @@ -882,7 +861,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } } - status = ie_core_load_network(ov_model->core, ov_model->network, ctx->options.device_type, &config, &ov_model->exe_network); + status = ie_core_load_network(ov_model->core, ov_model->network, ctx->device, &config, &ov_model->exe_network); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n"); status = ie_core_get_available_devices(ov_model->core, &a_dev); @@ -895,15 +874,15 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * APPEND_STRING(all_dev_names, a_dev.devices[i]) } av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all available devices are: \"%s\"\n", - ctx->options.device_type, all_dev_names); + ctx->device, all_dev_names); ret = AVERROR(ENODEV); goto err; } #endif // create infer_requests for async execution - if (ctx->options.nireq <= 0) { + if (ctx->nireq <= 0) { // the default value is a rough estimation - ctx->options.nireq = av_cpu_count() / 2 + 1; + ctx->nireq = av_cpu_count() / 2 + 1; } ov_model->request_queue = ff_safe_queue_create(); @@ -912,7 +891,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } - for (int i = 0; i < ctx->options.nireq; i++) { + for (int i = 0; i < ctx->nireq; i++) { OVRequestItem *item = av_mallocz(sizeof(*item)); if (!item) { ret = AVERROR(ENOMEM); @@ -945,7 +924,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } #endif - item->lltasks = av_malloc_array(ctx->options.batch_size, sizeof(*item->lltasks)); + item->lltasks = av_malloc_array(ctx->ov_option.batch_size, sizeof(*item->lltasks)); if (!item->lltasks) { ret = AVERROR(ENOMEM); goto err; @@ -994,7 +973,7 @@ static int execute_model_ov(OVRequestItem *request, Queue *inferenceq) LastLevelTaskItem *lltask; int ret = 0; TaskItem *task; - OVContext *ctx; + DnnContext *ctx; OVModel *ov_model; if (ff_queue_size(inferenceq) == 0) { @@ -1010,7 +989,7 @@ static int execute_model_ov(OVRequestItem *request, Queue *inferenceq) lltask = ff_queue_peek_front(inferenceq); task = lltask->task; ov_model = task->model; - ctx = &ov_model->ctx; + ctx = ov_model->ctx; ret = fill_model_input_ov(ov_model, request); if (ret != 0) { @@ -1084,8 +1063,8 @@ err: static int get_input_ov(void *model, DNNData *input, const char *input_name) { OVModel *ov_model = model; - OVContext *ctx = &ov_model->ctx; - int input_resizable = ctx->options.input_resizable; + DnnContext *ctx = ov_model->ctx; + int input_resizable = ctx->ov_option.input_resizable; #if HAVE_OPENVINO2 ov_shape_t input_shape = {0}; @@ -1291,7 +1270,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i #endif int ret; OVModel *ov_model = model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; TaskItem task; OVRequestItem *request; DNNExecBaseParams exec_params = { @@ -1308,7 +1287,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i } #if HAVE_OPENVINO2 - if (ctx->options.input_resizable) { + if (ctx->ov_option.input_resizable) { status = ov_partial_shape_create(4, dims, &partial_shape); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to create partial shape.\n"); @@ -1339,7 +1318,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i if (!ov_model->compiled_model) { #else - if (ctx->options.input_resizable) { + if (ctx->ov_option.input_resizable) { status = ie_network_get_input_shapes(ov_model->network, &input_shapes); input_shapes.shapes->shape.dims[2] = input_height; input_shapes.shapes->shape.dims[3] = input_width; @@ -1386,11 +1365,10 @@ err: return ret; } -static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; OVModel *ov_model = NULL; - OVContext *ctx = NULL; #if HAVE_OPENVINO2 ov_core_t* core = NULL; ov_model_t* ovmodel = NULL; @@ -1411,17 +1389,9 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f av_freep(&model); return NULL; } + ov_model->ctx = ctx; model->model = ov_model; ov_model->model = model; - ov_model->ctx.class = &dnn_openvino_class; - ctx = &ov_model->ctx; - - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - goto err; - } #if HAVE_OPENVINO2 status = ov_core_create(&core); @@ -1430,13 +1400,13 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f } ov_model->core = core; - status = ov_core_read_model(core, model_filename, NULL, &ovmodel); + status = ov_core_read_model(core, ctx->model_filename, NULL, &ovmodel); if (status != OK) { ov_version_t ver; status = ov_get_openvino_version(&ver); av_log(NULL, AV_LOG_ERROR, "Failed to read the network from model file %s,\n" "Please check if the model version matches the runtime OpenVINO Version:\n", - model_filename); + ctx->model_filename); if (status == OK) { av_log(NULL, AV_LOG_ERROR, "BuildNumber: %s\n", ver.buildNumber); } @@ -1452,13 +1422,13 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f if (status != OK) goto err; - status = ie_core_read_network(ov_model->core, model_filename, NULL, &ov_model->network); + status = ie_core_read_network(ov_model->core, ctx->model_filename, NULL, &ov_model->network); if (status != OK) { ie_version_t ver; ver = ie_c_api_version(); av_log(ctx, AV_LOG_ERROR, "Failed to read the network from model file %s,\n" "Please check if the model version matches the runtime OpenVINO %s\n", - model_filename, ver.api_version); + ctx->model_filename, ver.api_version); ie_version_free(&ver); goto err; } @@ -1496,7 +1466,6 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f model->get_input = &get_input_ov; model->get_output = &get_output_ov; - model->options = options; model->filter_ctx = filter_ctx; model->func_type = func_type; @@ -1510,7 +1479,7 @@ err: static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_params) { OVModel *ov_model = model->model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; OVRequestItem *request; TaskItem *task; int ret; @@ -1539,7 +1508,7 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOMEM); } - ret = ff_dnn_fill_task(task, exec_params, ov_model, ctx->options.async, 1); + ret = ff_dnn_fill_task(task, exec_params, ov_model, ctx->async, 1); if (ret != 0) { av_freep(&task); return ret; @@ -1557,8 +1526,8 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return ret; } - if (ctx->options.async) { - while (ff_queue_size(ov_model->lltask_queue) >= ctx->options.batch_size) { + if (ctx->async) { + while (ff_queue_size(ov_model->lltask_queue) >= ctx->ov_option.batch_size) { request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); @@ -1581,7 +1550,7 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOSYS); } - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { avpriv_report_missing_feature(ctx, "batch mode for sync execution"); return AVERROR(ENOSYS); } @@ -1604,7 +1573,7 @@ static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, static int dnn_flush_ov(const DNNModel *model) { OVModel *ov_model = model->model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; OVRequestItem *request; #if HAVE_OPENVINO2 ov_status_e status; @@ -1652,6 +1621,7 @@ static int dnn_flush_ov(const DNNModel *model) } const DNNModule ff_dnn_backend_openvino = { + .clazz = DNN_DEFINE_CLASS(dnn_openvino), .load_model = dnn_load_model_ov, .execute_model = dnn_execute_model_ov, .get_result = dnn_get_result_ov, diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 2ed17c3c87..d24591b90b 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -36,19 +36,8 @@ #include "safe_queue.h" #include -typedef struct TFOptions{ - char *sess_config; - uint8_t async; - uint32_t nireq; -} TFOptions; - -typedef struct TFContext { - const AVClass *class; - TFOptions options; -} TFContext; - -typedef struct TFModel{ - TFContext ctx; +typedef struct TFModel { + DnnContext *ctx; DNNModel *model; TF_Graph *graph; TF_Session *session; @@ -76,15 +65,13 @@ typedef struct TFRequestItem { DNNAsyncExecModule exec_module; } TFRequestItem; -#define OFFSET(x) offsetof(TFContext, x) +#define OFFSET(x) offsetof(TFOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_tensorflow_options[] = { - { "sess_config", "config for SessionOptions", OFFSET(options.sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, - DNN_BACKEND_COMMON_OPTIONS + { "sess_config", "config for SessionOptions", OFFSET(sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_tensorflow); static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue); static void infer_completion_callback(void *args); @@ -160,7 +147,7 @@ static int tf_start_inference(void *args) TFModel *tf_model = task->model; if (!request) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); + av_log(tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); return AVERROR(EINVAL); } @@ -170,7 +157,7 @@ static int tf_start_inference(void *args) task->nb_output, NULL, 0, NULL, request->status); if (TF_GetCode(request->status) != TF_OK) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); + av_log(tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); return DNN_GENERIC_ERROR; } return 0; @@ -198,7 +185,7 @@ static inline void destroy_request_item(TFRequestItem **arg) { static int extract_lltask_from_task(TaskItem *task, Queue *lltask_queue) { TFModel *tf_model = task->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; LastLevelTaskItem *lltask = av_malloc(sizeof(*lltask)); if (!lltask) { av_log(ctx, AV_LOG_ERROR, "Unable to allocate space for LastLevelTaskItem\n"); @@ -278,7 +265,7 @@ static TF_Tensor *allocate_input_tensor(const DNNData *input) static int get_input_tf(void *model, DNNData *input, const char *input_name) { TFModel *tf_model = model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TF_Status *status; TF_DataType dt; int64_t dims[4]; @@ -328,7 +315,7 @@ static int get_output_tf(void *model, const char *input_name, int input_width, i { int ret; TFModel *tf_model = model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TaskItem task; TFRequestItem *request; DNNExecBaseParams exec_params = { @@ -399,7 +386,7 @@ static int hex_to_data(uint8_t *data, const char *p) static int load_tf_model(TFModel *tf_model, const char *model_filename) { - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TF_Buffer *graph_def; TF_ImportGraphDefOptions *graph_opts; TF_SessionOptions *sess_opts; @@ -408,7 +395,7 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) int sess_config_length = 0; // prepare the sess config data - if (tf_model->ctx.options.sess_config != NULL) { + if (ctx->tf_option.sess_config != NULL) { const char *config; /* tf_model->ctx.options.sess_config is hex to present the serialized proto @@ -416,11 +403,11 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) proto in a python script, tools/python/tf_sess_config.py is a script example to generate the configs of sess_config. */ - if (strncmp(tf_model->ctx.options.sess_config, "0x", 2) != 0) { + if (strncmp(ctx->tf_option.sess_config, "0x", 2) != 0) { av_log(ctx, AV_LOG_ERROR, "sess_config should start with '0x'\n"); return AVERROR(EINVAL); } - config = tf_model->ctx.options.sess_config + 2; + config = ctx->tf_option.sess_config + 2; sess_config_length = hex_to_data(NULL, config); sess_config = av_mallocz(sess_config_length + AV_INPUT_BUFFER_PADDING_SIZE); @@ -461,7 +448,7 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) if (TF_GetCode(tf_model->status) != TF_OK) { TF_DeleteSessionOptions(sess_opts); av_log(ctx, AV_LOG_ERROR, "Failed to set config for sess options with %s\n", - tf_model->ctx.options.sess_config); + ctx->tf_option.sess_config); return DNN_GENERIC_ERROR; } } @@ -529,15 +516,14 @@ static void dnn_free_model_tf(DNNModel **model) TF_DeleteStatus(tf_model->status); } av_freep(&tf_model); - av_freep(model); + av_freep(&model); } } -static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; TFModel *tf_model = NULL; - TFContext *ctx = NULL; model = av_mallocz(sizeof(DNNModel)); if (!model){ @@ -551,23 +537,15 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f } model->model = tf_model; tf_model->model = model; - ctx = &tf_model->ctx; - ctx->class = &dnn_tensorflow_class; - - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - goto err; - } + tf_model->ctx = ctx; - if (load_tf_model(tf_model, model_filename) != 0){ - av_log(ctx, AV_LOG_ERROR, "Failed to load TensorFlow model: \"%s\"\n", model_filename); + if (load_tf_model(tf_model, ctx->model_filename) != 0){ + av_log(ctx, AV_LOG_ERROR, "Failed to load TensorFlow model: \"%s\"\n", ctx->model_filename); goto err; } - if (ctx->options.nireq <= 0) { - ctx->options.nireq = av_cpu_count() / 2 + 1; + if (ctx->nireq <= 0) { + ctx->nireq = av_cpu_count() / 2 + 1; } #if !HAVE_PTHREAD_CANCEL @@ -582,7 +560,7 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f goto err; } - for (int i = 0; i < ctx->options.nireq; i++) { + for (int i = 0; i < ctx->nireq; i++) { TFRequestItem *item = av_mallocz(sizeof(*item)); if (!item) { goto err; @@ -617,7 +595,6 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f model->get_input = &get_input_tf; model->get_output = &get_output_tf; - model->options = options; model->filter_ctx = filter_ctx; model->func_type = func_type; @@ -632,7 +609,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { LastLevelTaskItem *lltask; TaskItem *task; TFInferRequest *infer_request = NULL; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; int ret = 0; lltask = ff_queue_pop_front(tf_model->lltask_queue); @@ -728,7 +705,7 @@ static void infer_completion_callback(void *args) { DNNData *outputs; TFInferRequest *infer_request = request->infer_request; TFModel *tf_model = task->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; outputs = av_calloc(task->nb_output, sizeof(*outputs)); if (!outputs) { @@ -787,7 +764,7 @@ err: static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue) { TFModel *tf_model; - TFContext *ctx; + DnnContext *ctx; LastLevelTaskItem *lltask; TaskItem *task; int ret = 0; @@ -800,7 +777,7 @@ static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue) lltask = ff_queue_peek_front(lltask_queue); task = lltask->task; tf_model = task->model; - ctx = &tf_model->ctx; + ctx = tf_model->ctx; ret = fill_model_input_tf(tf_model, request); if (ret != 0) { @@ -833,7 +810,7 @@ err: static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TaskItem *task; TFRequestItem *request; int ret = 0; @@ -849,7 +826,7 @@ static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOMEM); } - ret = ff_dnn_fill_task(task, exec_params, tf_model, ctx->options.async, 1); + ret = ff_dnn_fill_task(task, exec_params, tf_model, ctx->async, 1); if (ret != 0) { av_log(ctx, AV_LOG_ERROR, "Fill task with invalid parameter(s).\n"); av_freep(&task); @@ -887,7 +864,7 @@ static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, static int dnn_flush_tf(const DNNModel *model) { TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TFRequestItem *request; int ret; @@ -915,6 +892,7 @@ static int dnn_flush_tf(const DNNModel *model) } const DNNModule ff_dnn_backend_tf = { + .clazz = DNN_DEFINE_CLASS(dnn_tensorflow), .load_model = dnn_load_model_tf, .execute_model = dnn_execute_model_tf, .get_result = dnn_get_result_tf, diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index ae55893a50..abdef1f178 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -36,18 +36,8 @@ extern "C" { #include "safe_queue.h" } -typedef struct THOptions{ - char *device_name; - int optimize; -} THOptions; - -typedef struct THContext { - const AVClass *c_class; - THOptions options; -} THContext; - typedef struct THModel { - THContext ctx; + DnnContext *ctx; DNNModel *model; torch::jit::Module *jit_model; SafeQueue *request_queue; @@ -67,20 +57,17 @@ typedef struct THRequestItem { } THRequestItem; -#define OFFSET(x) offsetof(THContext, x) +#define OFFSET(x) offsetof(THOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_th_options[] = { - { "device", "device to run model", OFFSET(options.device_name), AV_OPT_TYPE_STRING, { .str = "cpu" }, 0, 0, FLAGS }, - { "optimize", "turn on graph executor optimization", OFFSET(options.optimize), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS}, + { "optimize", "turn on graph executor optimization", OFFSET(optimize), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS}, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_th); - static int extract_lltask_from_task(TaskItem *task, Queue *lltask_queue) { THModel *th_model = (THModel *)task->model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; LastLevelTaskItem *lltask = (LastLevelTaskItem *)av_malloc(sizeof(*lltask)); if (!lltask) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for LastLevelTaskItem\n"); @@ -153,7 +140,6 @@ static void dnn_free_model_th(DNNModel **model) } ff_queue_destroy(th_model->task_queue); delete th_model->jit_model; - av_opt_free(&th_model->ctx); av_freep(&th_model); av_freep(model); } @@ -181,7 +167,7 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) TaskItem *task = NULL; THInferRequest *infer_request = NULL; DNNData input = { 0 }; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; int ret, width_idx, height_idx, channel_idx; lltask = (LastLevelTaskItem *)ff_queue_pop_front(th_model->lltask_queue); @@ -241,7 +227,7 @@ static int th_start_inference(void *args) LastLevelTaskItem *lltask = NULL; TaskItem *task = NULL; THModel *th_model = NULL; - THContext *ctx = NULL; + DnnContext *ctx = NULL; std::vector inputs; torch::NoGradGuard no_grad; @@ -253,9 +239,9 @@ static int th_start_inference(void *args) lltask = request->lltask; task = lltask->task; th_model = (THModel *)task->model; - ctx = &th_model->ctx; + ctx = th_model->ctx; - if (ctx->options.optimize) + if (ctx->torch_option.optimize) torch::jit::setGraphExecutorOptimize(true); else torch::jit::setGraphExecutorOptimize(false); @@ -292,7 +278,7 @@ static void infer_completion_callback(void *args) { outputs.dims[2] = sizes.at(2); // H outputs.dims[3] = sizes.at(3); // W } else { - avpriv_report_missing_feature(&th_model->ctx, "Support of this kind of model"); + avpriv_report_missing_feature(th_model->ctx, "Support of this kind of model"); goto err; } @@ -304,7 +290,7 @@ static void infer_completion_callback(void *args) { if (th_model->model->frame_post_proc != NULL) { th_model->model->frame_post_proc(task->out_frame, &outputs, th_model->model->filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, &outputs, &th_model->ctx); + ff_proc_from_dnn_to_frame(task->out_frame, &outputs, th_model->ctx); } } else { task->out_frame->width = outputs.dims[dnn_get_width_idx_by_layout(outputs.layout)]; @@ -312,7 +298,7 @@ static void infer_completion_callback(void *args) { } break; default: - avpriv_report_missing_feature(&th_model->ctx, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model->func_type); goto err; } task->inference_done++; @@ -322,7 +308,7 @@ err: if (ff_safe_queue_push_back(th_model->request_queue, request) < 0) { destroy_request_item(&request); - av_log(&th_model->ctx, AV_LOG_ERROR, "Unable to push back request_queue when failed to start inference.\n"); + av_log(th_model->ctx, AV_LOG_ERROR, "Unable to push back request_queue when failed to start inference.\n"); } } @@ -352,7 +338,7 @@ static int execute_model_th(THRequestItem *request, Queue *lltask_queue) goto err; } if (task->async) { - avpriv_report_missing_feature(&th_model->ctx, "LibTorch async"); + avpriv_report_missing_feature(th_model->ctx, "LibTorch async"); } else { ret = th_start_inference((void *)(request)); if (ret != 0) { @@ -375,7 +361,7 @@ static int get_output_th(void *model, const char *input_name, int input_width, i { int ret = 0; THModel *th_model = (THModel*) model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; TaskItem task = { 0 }; THRequestItem *request = NULL; DNNExecBaseParams exec_params = { @@ -424,12 +410,12 @@ static THInferRequest *th_create_inference_request(void) return request; } -static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; THModel *th_model = NULL; THRequestItem *item = NULL; - THContext *ctx; + const char *device_name = ctx->device ? ctx->device : "cpu"; model = (DNNModel *)av_mallocz(sizeof(DNNModel)); if (!model) { @@ -443,24 +429,17 @@ static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType f } th_model->model = model; model->model = th_model; - th_model->ctx.c_class = &dnn_th_class; - ctx = &th_model->ctx; - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - return NULL; - } + th_model->ctx = ctx; - c10::Device device = c10::Device(ctx->options.device_name); + c10::Device device = c10::Device(device_name); if (!device.is_cpu()) { - av_log(ctx, AV_LOG_ERROR, "Not supported device:\"%s\"\n", ctx->options.device_name); + av_log(ctx, AV_LOG_ERROR, "Not supported device:\"%s\"\n", device_name); goto fail; } try { th_model->jit_model = new torch::jit::Module; - (*th_model->jit_model) = torch::jit::load(model_filename); + (*th_model->jit_model) = torch::jit::load(ctx->model_filename); } catch (const c10::Error& e) { av_log(ctx, AV_LOG_ERROR, "Failed to load torch model\n"); goto fail; @@ -502,7 +481,6 @@ static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType f model->get_input = &get_input_th; model->get_output = &get_output_th; - model->options = NULL; model->filter_ctx = filter_ctx; model->func_type = func_type; return model; @@ -519,7 +497,7 @@ fail: static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_params) { THModel *th_model = (THModel *)model->model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; TaskItem *task; THRequestItem *request; int ret = 0; @@ -582,7 +560,7 @@ static int dnn_flush_th(const DNNModel *model) request = (THRequestItem *)ff_safe_queue_pop_front(th_model->request_queue); if (!request) { - av_log(&th_model->ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + av_log(th_model->ctx, AV_LOG_ERROR, "unable to get infer request.\n"); return AVERROR(EINVAL); } @@ -590,6 +568,7 @@ static int dnn_flush_th(const DNNModel *model) } extern const DNNModule ff_dnn_backend_torch = { + .clazz = DNN_DEFINE_CLASS(dnn_th), .load_model = dnn_load_model_th, .execute_model = dnn_execute_model_th, .get_result = dnn_get_result_th, diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index b9f71aea53..245fe5f759 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -25,11 +25,59 @@ #include "../dnn_interface.h" #include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavfilter/internal.h" extern const DNNModule ff_dnn_backend_openvino; extern const DNNModule ff_dnn_backend_tf; extern const DNNModule ff_dnn_backend_torch; +#define OFFSET(x) offsetof(DnnContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM +static const AVOption dnn_base_options[] = { + {"model", "path to model file", + OFFSET(model_filename), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"input", "input name of the model", + OFFSET(model_inputname), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"output", "output name of the model", + OFFSET(model_outputnames_string), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"backend_configs", "backend configs (deprecated)", + OFFSET(backend_options), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED}, + {"options", "backend configs (deprecated)", + OFFSET(backend_options), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED}, + {"nireq", "number of request", + OFFSET(nireq), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, FLAGS}, + {"async", "use DNN async inference", + OFFSET(async), AV_OPT_TYPE_BOOL, {.i64 = 1}, 0, 1, FLAGS}, + {"device", "device to run model", + OFFSET(device), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {NULL} +}; + +AVFILTER_DEFINE_CLASS(dnn_base); + +typedef struct DnnBackendInfo { + const size_t offset; + union { + const AVClass *class; + const DNNModule *module; + }; +} DnnBackendInfo; + +static const DnnBackendInfo dnn_backend_info_list[] = { + {0, .class = &dnn_base_class}, + // Must keep the same order as in DNNOptions, so offset value in incremental order +#ifdef CONFIG_LIBTENSORFLOW + {offsetof(DnnContext, tf_option), .module = &ff_dnn_backend_tf}, +#endif +#ifdef CONFIG_LIBOPENVINO + {offsetof(DnnContext, ov_option), .module = &ff_dnn_backend_openvino}, +#endif +#ifdef CONFIG_LIBTORCH + {offsetof(DnnContext, torch_option), .module = &ff_dnn_backend_torch}, +#endif +}; + const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) { switch(backend_type){ @@ -52,3 +100,36 @@ const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) return NULL; } } + +void *ff_dnn_child_next(DnnContext *obj, void *prev) { + size_t pre_offset; + char *ptr; + + if (!prev) { + obj->clazz = &dnn_base_class; + return obj; + } + + pre_offset = (char *)prev - (char *)obj; + for (int i = 0; i < FF_ARRAY_ELEMS(dnn_backend_info_list) - 1; i++) { + if (dnn_backend_info_list[i].offset == pre_offset) { + ptr = (char *)obj + dnn_backend_info_list[i + 1].offset; + *(const AVClass **) ptr = dnn_backend_info_list[i + 1].class; + return ptr; + } + } + + return NULL; +} + +const AVClass *ff_dnn_child_class_iterate(void **iter) +{ + uintptr_t i = (uintptr_t) *iter; + + if (i < FF_ARRAY_ELEMS(dnn_backend_info_list)) { + *iter = (void *)(i + 1); + return dnn_backend_info_list[i].class; + } + + return NULL; +} \ No newline at end of file diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 5e76b9ba45..3dd51badf6 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -19,6 +19,7 @@ #include "dnn_filter_common.h" #include "libavutil/avstring.h" #include "libavutil/mem.h" +#include "libavutil/opt.h" #define MAX_SUPPORTED_OUTPUTS_NB 4 @@ -52,6 +53,17 @@ static char **separate_output_names(const char *expr, const char *val_sep, int * return parsed_vals; } +typedef struct DnnFilterBase { + const AVClass *class; + DnnContext dnnctx; +} DnnFilterBase; + +void *ff_dnn_filter_child_next(void *obj, void *prev) +{ + DnnFilterBase *base = obj; + return ff_dnn_child_next(&base->dnnctx, prev); +} + int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNBackendType backend = ctx->backend_type; @@ -91,7 +103,25 @@ int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *fil return AVERROR(EINVAL); } - ctx->model = (ctx->dnn_module->load_model)(ctx->model_filename, func_type, ctx->backend_options, filter_ctx); + if (ctx->backend_options) { + void *child = NULL; + + av_log(filter_ctx, AV_LOG_WARNING, + "backend_configs is deprecated, please set backend options directly\n"); + while (child = ff_dnn_child_next(ctx, child)) { + if (*(const AVClass **)child == &ctx->dnn_module->clazz) { + int ret = av_opt_set_from_string(child, ctx->backend_options, + NULL, "=", "&"); + if (ret < 0) { + av_log(filter_ctx, AV_LOG_ERROR, "failed to parse options \"%s\"\n", + ctx->backend_options); + return ret; + } + } + } + } + + ctx->model = (ctx->dnn_module->load_model)(ctx, func_type, filter_ctx); if (!ctx->model) { av_log(filter_ctx, AV_LOG_ERROR, "could not load DNN model\n"); return AVERROR(EINVAL); diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h index 30871ee381..854b80fd53 100644 --- a/libavfilter/dnn_filter_common.h +++ b/libavfilter/dnn_filter_common.h @@ -26,28 +26,21 @@ #include "dnn_interface.h" -typedef struct DnnContext { - char *model_filename; - DNNBackendType backend_type; - char *model_inputname; - char *model_outputnames_string; - char *backend_options; - int async; - - char **model_outputnames; - uint32_t nb_outputs; - const DNNModule *dnn_module; - DNNModel *model; -} DnnContext; - -#define DNN_COMMON_OPTIONS \ - { "model", "path to model file", OFFSET(model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "input", "input name of the model", OFFSET(model_inputname), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "output", "output name of the model", OFFSET(model_outputnames_string), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "backend_configs", "backend configs", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "options", "backend configs (deprecated, use backend_configs)", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED},\ - { "async", "use DNN async inference (ignored, use backend_configs='async=1')", OFFSET(async), AV_OPT_TYPE_BOOL, { .i64 = 1}, 0, 1, FLAGS}, - +#define AVFILTER_DNN_DEFINE_CLASS_EXT(name, desc, options) \ + static const AVClass name##_class = { \ + .class_name = desc, \ + .item_name = av_default_item_name, \ + .option = options, \ + .version = LIBAVUTIL_VERSION_INT, \ + .category = AV_CLASS_CATEGORY_FILTER, \ + .child_next = ff_dnn_filter_child_next, \ + .child_class_iterate = ff_dnn_child_class_iterate, \ + } + +#define AVFILTER_DNN_DEFINE_CLASS(fname) \ + AVFILTER_DNN_DEFINE_CLASS_EXT(fname, #fname, fname##_options) + +void *ff_dnn_filter_child_next(void *obj, void *prev); int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); int ff_dnn_set_frame_proc(DnnContext *ctx, FramePrePostProc pre_proc, FramePrePostProc post_proc); diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 63f492e690..9516130714 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -93,8 +93,6 @@ typedef int (*ClassifyPostProc)(AVFrame *frame, DNNData *output, uint32_t bbox_i typedef struct DNNModel{ // Stores model that can be different for different backends. void *model; - // Stores options when the model is executed by the backend - const char *options; // Stores FilterContext used for the interaction between AVFrame and DNNData AVFilterContext *filter_ctx; // Stores function type of the model @@ -117,10 +115,65 @@ typedef struct DNNModel{ ClassifyPostProc classify_post_proc; } DNNModel; +typedef struct TFOptions{ + const AVClass *clazz; + + char *sess_config; +} TFOptions; + +typedef struct OVOptions { + const AVClass *clazz; + + int batch_size; + int input_resizable; + DNNLayout layout; + float scale; + float mean; +} OVOptions; + +typedef struct THOptions { + const AVClass *clazz; + int optimize; +} THOptions; + +typedef struct DNNModule DNNModule; + +typedef struct DnnContext { + const AVClass *clazz; + + DNNModel *model; + + char *model_filename; + DNNBackendType backend_type; + char *model_inputname; + char *model_outputnames_string; + char *backend_options; + int async; + + char **model_outputnames; + uint32_t nb_outputs; + const DNNModule *dnn_module; + + int nireq; + char *device; + +#ifdef CONFIG_LIBTENSORFLOW + TFOptions tf_option; +#endif + +#ifdef CONFIG_LIBOPENVINO + OVOptions ov_option; +#endif +#ifdef CONFIG_LIBTORCH + THOptions torch_option; +#endif +} DnnContext; + // Stores pointers to functions for loading, executing, freeing DNN models for one of the backends. -typedef struct DNNModule{ +struct DNNModule { + const AVClass clazz; // Loads model and parameters from given file. Returns NULL if it is not possible. - DNNModel *(*load_model)(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); + DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); // Executes model with specified input and output. Returns the error code otherwise. int (*execute_model)(const DNNModel *model, DNNExecBaseParams *exec_params); // Retrieve inference result. @@ -129,11 +182,14 @@ typedef struct DNNModule{ int (*flush)(const DNNModel *model); // Frees memory allocated for model. void (*free_model)(DNNModel **model); -} DNNModule; +}; // Initializes DNNModule depending on chosen backend. const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx); +void *ff_dnn_child_next(DnnContext *obj, void *prev); +const AVClass *ff_dnn_child_class_iterate(void **iter); + static inline int dnn_get_width_idx_by_layout(DNNLayout layout) { return layout == DL_NHWC ? 2 : 3; diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index c8848dd7ba..f47019a3b4 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -46,13 +46,10 @@ static const AVOption derain_options[] = { #if (CONFIG_LIBTENSORFLOW == 1) { "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, .unit = "backend" }, #endif - { "model", "path to model file", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, - { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(derain); +AVFILTER_DNN_DEFINE_CLASS(derain); static int filter_frame(AVFilterLink *inlink, AVFrame *in) { diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index 1f8f227e3a..f863f7fcff 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -50,14 +50,13 @@ static const AVOption dnn_classify_options[] = { #if (CONFIG_LIBOPENVINO == 1) { "openvino", "openvino backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" }, #endif - DNN_COMMON_OPTIONS { "confidence", "threshold of confidence", OFFSET2(confidence), AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS}, { "labels", "path to labels file", OFFSET2(labels_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { "target", "which one to be classified", OFFSET2(target), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_classify); +AVFILTER_DNN_DEFINE_CLASS(dnn_classify); static int dnn_classify_post_proc(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx) { diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index bacea3ef29..cdfa355ef0 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -70,7 +70,6 @@ static const AVOption dnn_detect_options[] = { #if (CONFIG_LIBOPENVINO == 1) { "openvino", "openvino backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" }, #endif - DNN_COMMON_OPTIONS { "confidence", "threshold of confidence", OFFSET2(confidence), AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS}, { "labels", "path to labels file", OFFSET2(labels_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { "model_type", "DNN detection model type", OFFSET2(model_type), AV_OPT_TYPE_INT, { .i64 = DDMT_SSD }, INT_MIN, INT_MAX, FLAGS, .unit = "model_type" }, @@ -85,7 +84,7 @@ static const AVOption dnn_detect_options[] = { { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_detect); +AVFILTER_DNN_DEFINE_CLASS(dnn_detect); static inline float sigmoid(float x) { return 1.f / (1.f + exp(-x)); diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index fdac31665e..ed6ad1f959 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -54,11 +54,10 @@ static const AVOption dnn_processing_options[] = { #if (CONFIG_LIBTORCH == 1) { "torch", "torch backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_TH }, 0, 0, FLAGS, "backend" }, #endif - DNN_COMMON_OPTIONS { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_processing); +AVFILTER_DNN_DEFINE_CLASS(dnn_processing); static av_cold int init(AVFilterContext *context) { diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 60683b5209..e33a3b4641 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -50,13 +50,10 @@ static const AVOption sr_options[] = { { "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, .unit = "backend" }, #endif { "scale_factor", "scale factor for SRCNN model", OFFSET(scale_factor), AV_OPT_TYPE_INT, { .i64 = 2 }, 2, 4, FLAGS }, - { "model", "path to model file specifying network architecture and its parameters", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, - { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(sr); +AVFILTER_DNN_DEFINE_CLASS(sr); static av_cold int init(AVFilterContext *context) { From patchwork Sat Apr 27 16:41:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48297 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1069863pzb; Sat, 27 Apr 2024 09:42:37 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVnt0HcKKVpEFtdZwNERgBBJfg7kFgXQKoNFTEyliYSih8ewwcQr4t8q5t+Kw0oosbLxpKCEug95AIUE0CIVws31ph3JN2IKqYmWw== X-Google-Smtp-Source: AGHT+IHQIninM0qPnpYqaObaJhM5njMcFINd/uJfMTr96eh88m0vt3FucvcA8MSfII97GUFMAJ7l X-Received: by 2002:a2e:920e:0:b0:2da:320a:6739 with SMTP id k14-20020a2e920e000000b002da320a6739mr3720507ljg.1.1714236156700; Sat, 27 Apr 2024 09:42:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236156; cv=none; d=google.com; s=arc-20160816; b=S8YCZhw7a7hIIYOPA7/d5ufPCVs9p8tZTbrIWrRInTKRX2AhtF8KReS+vC1D1Nb9ZD Sxaf4M8qieF0mPJ50ppO3i1WLW865Dql9r1mGiXJQy/BBVoMrqINff0ng5xfYO1RkxyG nRBWRMgzNOxlk9KVCm2qcKgeamumPLOs0Jbe1uI+v5ZaoXG76vzvp0JdQ4npe2OvxnjB bCYs+/JUo/QLKA6+2Q0wekAr2o2tXWTBXgaayn2N14H1lkRktMfpq1sFJJQgGIaZqHwi BcMqBnyDsAL834bm5NgBgx1nxaTAUYhvU0cTtTseF6o/xmvNO/NA+gSQG1gdcjE4xyFh h/8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=Z7qGnFMm0dw2V0FIpE8QvtaCl8o9G0F6TZQlNPSbj9Y=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=EUD/uZc4D7FkNYi6/MUg7su8zqrK7P5w7cmN+ApKBraa2rf2KHnbWG/kb3VhoWL//F lmJs1y+PvPxyMMS9Bs6XMTTEYy1pEPCszq6R+rAeoD8n19zutAN6vHizf1KPlxiRX+y9 32oxnwci7ScF8iPoStbatrAwOtmvYSJkykwqXWFqrv5DHcJPXPPJRfjjQr2hjzWtfrYj XPUAErx7WREtIAm3mRijRJ8TO+5LxRVkhYOtOKGPa6qQ8O6cq41is1CzvRvIeeT/hn/C mlIDlpkdjjfQ8UF72+e+4I5dvAw8oFJLZLaih6Yw3ScU/qtHs79u/VcBoJ7Df7Gdk0IC lijw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=nb3cmAIi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q19-20020a2e8753000000b002de846909ccsi2604061ljj.70.2024.04.27.09.42.36; Sat, 27 Apr 2024 09:42:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=nb3cmAIi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 89E6268D2B0; Sat, 27 Apr 2024 19:42:22 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-216.mail.qq.com (out162-62-58-216.mail.qq.com [162.62.58.216]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ADBE868D220 for ; Sat, 27 Apr 2024 19:42:13 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236124; bh=U20Oc3roQVVwUry5FoOLmCq7mAoTmM8HL35RBZB8e98=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=nb3cmAIiyEXpkOPEqXCcQ7vvEqHyzG/7FT9ECDjTs2Pe2096Yh4wMnYgsYtwMIFPv +D8RnEchEn3i+95f3trDaQeRZmQzqvVaDVD9q3SgshNveLt3aGFAyrg1dFTsVht03K /0DmraoguAzL1rJaphrdpdJS0dvQoTzuaaJIMd/Q= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236123tzvco1cnq Message-ID: X-QQ-XMAILINFO: MRw/zKT/0BpPlSRBL1Q+7bzAumsrtxdwMj4/untB2UVealJVvoDoG6IXWPNNMW QYkeJvL1uIxH6EbcWuMX2MY9Ml62hwq94iNEqkWOHLsUFWp0yTeAEp91qwSDFjuUzieG/5dM9B7J HqHkxZ8NTTVT2TTwkYuUJ67gu/aLc3fmcBOTt7WNEuKg6qDWBoBVxgE5w7qETqD/RG4E7CFvrSWU i0L7Net05Ipb5dk9APiPM+/p+JawY2NNV8EASHjOND0TeD2bFOQvrU84ILcCBmijM4F8kD+iymZ0 gPCrGg1J0MO0jGeRln415Rc8T47vtDaCwW2v2vPhZ+DeERaErluAC1jlZ4IamWYkeVnB219SEfZC 5jEqZH89PSlJb8OS/+pUxHXu1eLBqYecoKarnYcE1hp06pvY8BrIDF59JWCXtqnuM/xWJxTeB6Ea eBobnBE0pqRzcV7wUGOaIJX2m30isvgX9ST5qF2hXXWIMftrNGOcdVByOn+8skfl89Kc8peOnYUl iYE0wi+rmHBXZF9NeK/e2Ms6UhaQuxJ1RnwjElUesGgCuJjCy9iR0nS1TPXe2XFsPeAV/uWg5G6I XeYF6od6Zhe0KE+2UODMkcboUGe/3w7bYZ0zl7ey/RzKm1YKP5Mbo4eeAQ6ExCOp4m+EqYc32cn4 TbgxTCuh2U5f22MA4E7tMvuwJoAGHD9VNnkRu2syyJXyD6elmsk5Dj7BNikiXTyfyvEZw1s2S5YH 2Vqb+7z6oJwKv8qmkzH6LEV3782cOOfw+JaOthktqE6mVmieRvrmb+MMdjnp6AtXey4aTRe/rpH5 6T51mYIktzqbTjh3jfke+lD871d/12HPNzk1JAzDN6gK/awpBrCLxCv8XkwlJk2HAgq9AIjK4S1/ iFC4Tj2JO7skAa9g948+Wx08CwjMAoIKJUEowYGoSmMZQhWJBHJ5T5rrNp7yQZGBNwKUrtfJ5usS HO/I0Zd39YbkxDj5RGhDeTur0BJTWlDBdB4qBqpVwh3aIu2cWCQrgr4sPt4X0ZFtjkIel0C/M= X-QQ-XMRINFO: M/715EihBoGSf6IYSX1iLFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:52 +0800 X-OQ-MSGID: <20240427164159.82771-3-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 2/9] avfilter/dnn_backend_openvino: Fix free context at random place X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ejNQzW2GmPrk From: Zhao Zhili It will be freed again by ff_dnn_uninit. --- libavfilter/dnn/dnn_backend_openvino.c | 1 - 1 file changed, 1 deletion(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index c4b0682f11..769ba0a54b 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -959,7 +959,6 @@ err: if (input_model_info) ov_preprocess_input_model_info_free(input_model_info); #endif - dnn_free_model_ov(&ov_model->model); return ret; } From patchwork Sat Apr 27 16:41:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48300 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070062pzb; Sat, 27 Apr 2024 09:43:07 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVGPnZOm+HkbYfObafiw9NzExzvKgJcQCu5FFxhIKj6VXa3h50BEK3++RSp9y/d12t7cfbAwrEfnr3cBwuZNDL3j3LF8EdUPs4STQ== X-Google-Smtp-Source: AGHT+IEbjyOQYVg4lAWsDqWlW9vIFfTDoU6il3hAGbTJerJ6Tv54t9u6uxeZJEugtSEp4I2CPDdP X-Received: by 2002:a05:651c:b0c:b0:2e0:510:a498 with SMTP id b12-20020a05651c0b0c00b002e00510a498mr303961ljr.31.1714236187148; Sat, 27 Apr 2024 09:43:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236187; cv=none; d=google.com; s=arc-20160816; b=lrHmzf9GjEZZFuhv11Elv2Uc7myHd7hgxEUnrSQaBwNlT9mLW03uPxyWwJ5UdohmoP 4GvLO8qdu7wM+hglcqjogD6U2Y7oWyt6x/X107V8LjbBwF4jJZJFiZFHaN2sw2kI5Miu u56DBim8gC+UA656JANpZi8yOsFbiaLTBD8b8O92ChPgupx56QdPYJ2BNIgzd9pvj+Z0 jaFTzzM3y/J+vQIzlZG7Nc36+lN7RVfssGEr/8Eh0TYmJwxrusQpHmdjFlpFtQoxksJs U0EuEKhOs1h6hmen6XZCTBRgFH/AGTTbkQLem5qlGgOKca14h0Rqk/Ag8lpe37r/i7cB em+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=yZb+rWBxBsX0/b60vnoAgSeZaHH2+5e3TNxE/jMWeB4=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=eXeVd2FHRUBfzQ/PWvqesP5ZuQgnEoERazzRxDU83LeTeetfGdWIKGukOBOuHeRLVi UAdDrE1xFkWX6RT4DA0sUoYvWRVeE6BhEHcg5ldvFDsmauMWvT0ilGcKk6xDyEqPFKgr ArSxVRVkq3D5UzeakzsVfFHZeqfrM4vMwFJIRp9f3xLQFBqcNEgKiShlTFPskHVcmPSa anw+iQhL2hXWLVW5B5FHeg5PT0D86mcRuu9hpHelbnzySh0SEzVdVsGBQWKBRZhkMt+U ijKGtua0D8QwovZi0qyE7W1fpL7OUfMYhZkO23n21ciNmOGaE9xWoPKcuTbAPQU6mHAx rv+Q==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=IU6dPQaB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c13-20020a2ea1cd000000b002df1b07a7bdsi1528517ljm.331.2024.04.27.09.43.06; Sat, 27 Apr 2024 09:43:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=IU6dPQaB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EE72868D2F2; Sat, 27 Apr 2024 19:42:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-73.mail.qq.com (out203-205-251-73.mail.qq.com [203.205.251.73]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E8C7168D2A7 for ; Sat, 27 Apr 2024 19:42:14 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236124; bh=o/PnOJjdVuY5vx6DWP3SJIn5JrYEbjSnkPSAyGPUkoM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=IU6dPQaBSEc41AA+m5/VP6T/gY3IxorZ0+bg82G4bKY9wTYz8kYjKFfKtwFgGtmBj ZfCQYezaeDvFolUtkTnLVpmBwRgHqLZtC0eHg3bNsZFWBYWlmFh4+Ej5DGpZr+qg5E WqEGSCBxhtRxwQg9+o8S8dPEmI0D3dZqDJaTcdFI= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236124tg0lufeca Message-ID: X-QQ-XMAILINFO: OKKHiI6c9SH39yEiQZBt0jasuuodqk0X8PmCh0e9prX9PGTHDADS/bu22JVKVr 5EOCgdAq5Z+56Tua44OfBegMi0y4ZJ3YVlwHz3ZlSEqCZAh6cOGQBZoQ7MXgXJYyU+Xg+EGLsYva WwgY82LI1pn9EMAO1iqLWTfd2OOwt8KJdtQVwycJlw057tZPt4hdPJErCUbhq1qSW+4LOZikRBmk xXxXQzKJdVdHxYyxqZUS7Yp7DDCmnFkbS+5p9HHXu6q/XGBo6mYs96nFuof/bfMZgRhy4KRuKF3c PW9MvIXcttnB3kOLk5JO8plivZGFyfp9NmnLYafrdBtyEw3yepeT5hecGg4tYZMUW9WP3UIXZGfl iauxpbce6p2q6kC/dV/i3wPpG8SkxjaW8McxkXyETliukM44TN1cKTuiwhrVK3WUEIbvLl/3TmT5 parEG1pMubhUSY5vc//jo9DzjPLqihgsVW7SrWpO3QayMPfl6K/ciOe+PUu5NDbSm+N1MV5jUep+ yXB1rEZbc1HipWvjy4G2muMpTy5qcs4VyTbluR8LVqAjvmhRhpCyZ2tT7Dkxg7RxoTQFgahzSCyM 2OietgHcUY+fEMgdfYHhuHPxRpCH1Boqhu/PEj3MvB6ovhoUSHfvk8U/NutLIew/eXm7m1c494/I e1m/Xhhybeh6B5oBYUIy+GLJ+CeS0fqDdHaF4RdEk8knVxzFN5JjnVYJ/sm6TntlxRNX4fz7qG8I skrLzJT/l02fNPrYVHwRThHuNXPZMrg4nDgAOk1ZYm+i3IEGi3N2VD4VxcNVoGImqHSnHTi2Ih58 5sEoA0RE5UYJ5FPZLuV/TMnunZmTYZY+1L4m5wDMnEiIONFfLb84CcseuOH6UT+Pg9o+EDLAT7Hy 5n++DMMsUk3Iu6hlwuuFnhnVaSjCXcnZR4tcOhLjZ+H9WXhON6q+2WozCAbGPdUpuVMuN4txsw0z qZYDrvyX+IQapt3qVEXsBsYeJ145e0zWnmIZiumap/vLc547zjYw== X-QQ-XMRINFO: NyFYKkN4Ny6FSmKK/uo/jdU= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:53 +0800 X-OQ-MSGID: <20240427164159.82771-4-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 3/9] avfilter/dnn_backend_openvino: simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ewG7TRsVmh28 From: Zhao Zhili --- libavfilter/dnn/dnn_backend_openvino.c | 47 +++++++++++--------------- 1 file changed, 20 insertions(+), 27 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 769ba0a54b..1acc54b791 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -41,8 +41,8 @@ #include "dnn_backend_common.h" typedef struct OVModel{ + DNNModel model; DnnContext *ctx; - DNNModel *model; #if HAVE_OPENVINO2 ov_core_t *core; ov_model_t *ov_model; @@ -300,11 +300,11 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) return ov2_map_error(status, NULL); } #endif - switch (ov_model->model->func_type) { + switch (ov_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (ov_model->model->frame_pre_proc != NULL) { - ov_model->model->frame_pre_proc(task->in_frame, &input, ov_model->model->filter_ctx); + if (ov_model->model.frame_pre_proc != NULL) { + ov_model->model.frame_pre_proc(task->in_frame, &input, ov_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } @@ -442,11 +442,11 @@ static void infer_completion_callback(void *args) for (int i = 0; i < request->lltask_count; ++i) { task = request->lltasks[i]->task; - switch (ov_model->model->func_type) { + switch (ov_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (ov_model->model->frame_post_proc != NULL) { - ov_model->model->frame_post_proc(task->out_frame, outputs, ov_model->model->filter_ctx); + if (ov_model->model.frame_post_proc != NULL) { + ov_model->model.frame_post_proc(task->out_frame, outputs, ov_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } @@ -458,23 +458,23 @@ static void infer_completion_callback(void *args) } break; case DFT_ANALYTICS_DETECT: - if (!ov_model->model->detect_post_proc) { + if (!ov_model->model.detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); goto end; } - ov_model->model->detect_post_proc(task->in_frame, outputs, + ov_model->model.detect_post_proc(task->in_frame, outputs, ov_model->nb_outputs, - ov_model->model->filter_ctx); + ov_model->model.filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: - if (!ov_model->model->classify_post_proc) { + if (!ov_model->model.classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); goto end; } for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++) - ov_model->model->classify_post_proc(task->in_frame, outputs, + ov_model->model.classify_post_proc(task->in_frame, outputs, request->lltasks[i]->bbox_index, - ov_model->model->filter_ctx); + ov_model->model.filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -571,7 +571,7 @@ static void dnn_free_model_ov(DNNModel **model) av_free(ov_model->all_input_names); #endif av_freep(&ov_model); - av_freep(model); + *model = NULL; } @@ -598,7 +598,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * #endif // We scale pixel by default when do frame processing. if (fabsf(ctx->ov_option.scale) < 1e-6f) - ctx->ov_option.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; + ctx->ov_option.scale = ov_model->model.func_type == DFT_PROCESS_FRAME ? 255 : 1; // batch size if (ctx->ov_option.batch_size <= 0) { ctx->ov_option.batch_size = 1; @@ -702,7 +702,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ret = ov2_map_error(status, NULL); goto err; } - if (ov_model->model->func_type != DFT_PROCESS_FRAME) + if (ov_model->model.func_type != DFT_PROCESS_FRAME) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); else if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); @@ -1280,7 +1280,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i .out_frame = NULL, }; - if (ov_model->model->func_type != DFT_PROCESS_FRAME) { + if (ov_model->model.func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); return AVERROR(EINVAL); } @@ -1342,7 +1342,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i goto err; } - ret = extract_lltask_from_task(ov_model->model->func_type, &task, ov_model->lltask_queue, NULL); + ret = extract_lltask_from_task(ov_model->model.func_type, &task, ov_model->lltask_queue, NULL); if (ret != 0) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); goto err; @@ -1378,19 +1378,12 @@ static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, A IEStatusCode status; #endif - model = av_mallocz(sizeof(DNNModel)); - if (!model){ - return NULL; - } - ov_model = av_mallocz(sizeof(OVModel)); - if (!ov_model) { - av_freep(&model); + if (!ov_model) return NULL; - } ov_model->ctx = ctx; + model = &ov_model->model; model->model = ov_model; - ov_model->model = model; #if HAVE_OPENVINO2 status = ov_core_create(&core); From patchwork Sat Apr 27 16:41:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48302 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070173pzb; Sat, 27 Apr 2024 09:43:26 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWnMNRrwPy95T/YjJegVHcEgrO6wkeNdeinzqlo2J9tQNWXzZB5tBO+ynH8wTr0fanJD1AiY203PDd8WxYrQlvcfLX8H1g8pNDqtg== X-Google-Smtp-Source: AGHT+IEtcR4SnWCcdB2XazM5StvntXd5ybkjrvYBb5w3uut5ncgN/yo0IXlisIdhpHSdGPyHaW1F X-Received: by 2002:a50:f69e:0:b0:572:7b2b:d96a with SMTP id d30-20020a50f69e000000b005727b2bd96amr76954edn.16.1714236206331; Sat, 27 Apr 2024 09:43:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236206; cv=none; d=google.com; s=arc-20160816; b=fQZIaL7Z2kXrfFbdmmDjc8I/X7d8BQeKifYcZA9FmjExP/Oh4DnZt7d6o/wOnyMcfI 9SXkkh7CfY8TnyCEA9IsfS234+Iv3acsJZWeb2Rwvc+y4UUUgixbG1GDl+j5zBw/s8yR +lYfIYw7dUZcam4w2ugAx+oBTA+ZFD6X2/BiU3x8N/UHPhMT6oAVnlxKfmI+SIy0sK9I w/gbw8VyJ1Sx/zY5X7ERiOA4y5ijLG4D7Zip/kWllPsXfw4T5jEKzAOG1M96STV6bzS6 rnTTPnGuxOrBkamb2viNBTLjq76KPsb7e4edPketyZvS40BPDma1qPiBlmFsRGllUErF QKAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=NcKvnJKy5P1JY93GJJGukbJqA5i2KQRq3fnAHueqrIo=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=maWaYsDG4lK+msTr3FX4rSzfhOrUuoh91OloE3D0P6XEOhheLLQvSSQmSDUNoJ3V5Q dooezrOw6AdqcZYiIH/H2qftJRLcRVAYEqvlcd8EgRNAWtVeRyWz9LP3OE/VsZRfdSSk i/ZCpyzlTZ9rMbbHfHkmFhxseMBZErWPAbUbHosa9JHOdiXCNlwUouGoVtGmc/ubTor/ RzbR5yj/eiSauF7hqjHtyPgHDsk6NrtuZGnR+g7JnBfydSq92lCdEZBWtytO8jlY8EC1 erlu9zuTSWMt5YkcFtXdIhB5ZecNriajlMG5bSb0hhsD0L6bMoCLOfuxWItVanintHks jClA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=LfQH5xNy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a15-20020a05640213cf00b005726b196fb9si1067454edx.510.2024.04.27.09.43.26; Sat, 27 Apr 2024 09:43:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=LfQH5xNy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E0A8968D405; Sat, 27 Apr 2024 19:42:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-210.mail.qq.com (out203-205-221-210.mail.qq.com [203.205.221.210]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 012E668D2AB for ; Sat, 27 Apr 2024 19:42:14 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236125; bh=ij3BuPB+1SDO4TrQLufpRme6GURUZXSvBIfypWPgQBQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LfQH5xNyw/Fu9XHG7thLHFNCJnDhNdpz6ycLOPHYG4mSWJ+uAD+Eg/uKTIVtnUtHt C9gL8N9VF6Z3iy3YyII8wXjXEOb5BWToY/D6YHyMwkPdbsaykheKW/EdkZfSkqaTf2 pbcmonFWDJRnUbOtucAzCVwLhJI98jrizqdPgqU8= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236124tv0wjsslv Message-ID: X-QQ-XMAILINFO: MK5U7QanZrCwSEL0746FVDmNNenX6OzdoJquzX76WyhSRMPmoBYA3ZYzkdc9Vl A1kVEWV/dTY86wlbCYaM/UDNaN7vpmLbCfBSSTDAHoAZy3XH5gbh7zvcnYTeVJPiVpyIAwnToS3B J6e+ius09cxxuQu8d0V5u4WlubiSK3MjVpewxrcBKliwl5HjuhoZevKcZfHTcUcqQ/UfxwG3r9JX bz0dC98C5FD1zkpGBUhZqIMF3SNew6DU29XBHUa8W62tL5FCiur7Vl6smI64y9H/22CAslbhHgpz RVWELCeDee1nn5O/jhx1vSLiag6MO7XijEo5FgZeFh8hVtseFINPlbytHvDJ5urkvA0jnBX/uy2J YG4nONK0uFLPoj+xnjbmrcZKHFk2P1xjZSoGPca6auoUdmE2jxcbeHHj0FGFK18Pnz4F+xNN8LE7 c7RMACo1Yl7JM9ZwYprQvSqoanEbjexLsKnhYweDjOvyOVz5ghwbrZznk27mtKN6tUp6k1HGF9Al 91hxaDpW7c/0cC/JJ5b9fvkj0Iu6OlmdUoAvPx1y8Y3Bi0zbOstztfxKKdneb0L/Scu9Ho6hq6WT 876oX/D7p2nHv17Sg1oM+/++BT54IkwYM1ZO05a2hErKrVBWEyePl/kLyNQ9EMoRNR7v0XgqWOlf TK4JEneugJPp+NHun8yg3703s3OdRUMMF5csdiHf2AcKvjRqwOmsAVCIH6LcsVR9NJ4PxUIHkMQ7 d2YCmcrAASvZVJQMa1SBRjswKTWNlNvPhg6dqyZ4fS/5uoGD5v4q1OBmNKP3iQwWrMyRnfdIJNVt bxXV0tu89yzvj0hw5obB3+Tvvohc81NCqSScVtivhAIMVjTc/Xp7HeTjPAdMbjtCipP6Gfpu+x1d jfHYQqLaQiSjsrly9gBOj1F5VJ5Sz8Sxsm2A3r6jrVWna7tV2dV3kFG/SsAwf6iayen28iacqYGT Zciszu896TB6Aj8iYM/LD1EZdoT4jCYWWEq4xptVg= X-QQ-XMRINFO: OWPUhxQsoeAVDbp3OJHYyFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:54 +0800 X-OQ-MSGID: <20240427164159.82771-5-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 4/9] avfilter/dnn_backend_tf: Remove one level of indentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nUmJ7bo4n1cT From: Zhao Zhili --- libavfilter/dnn/dnn_backend_tf.c | 63 ++++++++++++++++---------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index d24591b90b..60f9e57fb7 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -483,41 +483,42 @@ static void dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; - if (*model){ - tf_model = (*model)->model; - while (ff_safe_queue_size(tf_model->request_queue) != 0) { - TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); - destroy_request_item(&item); - } - ff_safe_queue_destroy(tf_model->request_queue); + if (!model || !*model) + return; - while (ff_queue_size(tf_model->lltask_queue) != 0) { - LastLevelTaskItem *item = ff_queue_pop_front(tf_model->lltask_queue); - av_freep(&item); - } - ff_queue_destroy(tf_model->lltask_queue); + tf_model = (*model)->model; + while (ff_safe_queue_size(tf_model->request_queue) != 0) { + TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + destroy_request_item(&item); + } + ff_safe_queue_destroy(tf_model->request_queue); - while (ff_queue_size(tf_model->task_queue) != 0) { - TaskItem *item = ff_queue_pop_front(tf_model->task_queue); - av_frame_free(&item->in_frame); - av_frame_free(&item->out_frame); - av_freep(&item); - } - ff_queue_destroy(tf_model->task_queue); + while (ff_queue_size(tf_model->lltask_queue) != 0) { + LastLevelTaskItem *item = ff_queue_pop_front(tf_model->lltask_queue); + av_freep(&item); + } + ff_queue_destroy(tf_model->lltask_queue); - if (tf_model->graph){ - TF_DeleteGraph(tf_model->graph); - } - if (tf_model->session){ - TF_CloseSession(tf_model->session, tf_model->status); - TF_DeleteSession(tf_model->session, tf_model->status); - } - if (tf_model->status){ - TF_DeleteStatus(tf_model->status); - } - av_freep(&tf_model); - av_freep(&model); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + + if (tf_model->graph){ + TF_DeleteGraph(tf_model->graph); + } + if (tf_model->session){ + TF_CloseSession(tf_model->session, tf_model->status); + TF_DeleteSession(tf_model->session, tf_model->status); + } + if (tf_model->status){ + TF_DeleteStatus(tf_model->status); } + av_freep(&tf_model); + av_freep(&model); } static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) From patchwork Sat Apr 27 16:41:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48301 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070130pzb; Sat, 27 Apr 2024 09:43:18 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUTJsG4KWW1m+wlxv2Zq+sVtlIzdHtNAXE/wo00lW7M2oulUOOw0HfFyJpXg4XI/7E4hCdQkaerKdEYskDatsaMWocOfH9RHnV7fA== X-Google-Smtp-Source: AGHT+IFXj4G6zsNW0z3HqqvTmJx08CxZ9rhPXaFGtzzg53Z2SwOgTCRjCM0Zahqczjqy9eheRhNG X-Received: by 2002:a05:6402:f24:b0:56f:e715:d1c7 with SMTP id i36-20020a0564020f2400b0056fe715d1c7mr4269568eda.1.1714236197755; Sat, 27 Apr 2024 09:43:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236197; cv=none; d=google.com; s=arc-20160816; b=MF1c5JbeqLL6LX2lLxLwaAg+jqEpmiq1Y/dRCkkPdYvwzVCV/Vh2moaXOF63J6afZy jEukrvSktmG1Y852ljQrHpHvOvYzTqlrG1mZ0gA/35yukRF3aMIATQ2XRI35OHBuHeoT 7+mw7P9S3acxj74ObsK54R7AFnF+1fd8bkqw1R0R6Dwty1t6vastSDCrTh2AGqZ78bLW 2hvgTe8+enTFLYRts08FCZo1E/pZDspAdPMOoH9MfF6+ntKx80prH3BGBkcoKUAomRQa 6sjbMnsWaWrdla5BO7Ki9hYM3/t4c2tmWr1xX5w7HjAOzscgIv04XnKFZIoLCAQgcu4U xStQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=zL/hXG5bouHUfUXNcgmG+JQtZ9mpVrmmG3Q9kO3rCCE=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=0msyHfsLxNmnPoGrYyuTiWn+Qs+JOBb3jWS/ohk27nzGMe59wNHJmhZ7RGybE9MJEp yOTqscIgz3p1w+3wM2QjAINfB2o7W2dFUFoDGMuKvwV3XewbyQAxzpSm4cHcVMfyl0RN 9z3IwDc0HH/lWwO4HMmRtTGjUel8/M93umkOKrlxnA+XGS40fFb6QD1J9ciMgT5zTP1k iwAx9ED/YYPOcCf2h6CnNVazoYX+6WSLxdmYy9SXqfFPYHeYJeu7iWeJChXaf3dwjJ6/ ZRIB+QT9lC9jH9Po1k5twGudbCBlIgzpoJgGcYaf7x7DvuVw9HyfiJJ2M4ebO9R8rUdi iC1A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=seP0oAoy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r24-20020a50d698000000b00571e5edc719si10186660edi.458.2024.04.27.09.43.16; Sat, 27 Apr 2024 09:43:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=seP0oAoy; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4267268D3F8; Sat, 27 Apr 2024 19:42:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-236.mail.qq.com (out203-205-221-236.mail.qq.com [203.205.221.236]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E266868D294 for ; Sat, 27 Apr 2024 19:42:14 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236125; bh=qkwweKsTJRKs0cDmDxI/i2ACwHQF2A6iSgS90qmmm1E=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=seP0oAoyQC5n7Ws80Y1MDhpx3FGeKEM2wJIs/czT9gx5W+sAMgy+vEzkC+R4wjmdD LGPEW8/plLCzGCDGxZ8OCilAg+wMJEynO8rbMH5u5bCShNqO/K1gIihaCyQ5B6ymRq yBUWBZUYu5q6Kuya9walEEDfdSZXWokcy512KYoQ= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236125tmo0r3es5 Message-ID: X-QQ-XMAILINFO: OACZwgAjMvjH/Q0ne5n/2C1ebwq76rI5yolSO47saa7Bxv4yZXAWDz122ocncQ VBr4Ks+RPa+W9Jwp7OKf6PkUrH+aY4LuK8RowhpPRw4mRn8Q3O4H5abd7/I+mEH3tqfnvbQZLEI/ LLbK5Nna+HaQy2crwIiEW5PXEbPwQLNN94HJiTJ7KxiZ934Km3Gl+N9+1rf3uwGhHlsMhMjo+COf BO/Poc7wGQs/5orKpyG+gAJ/F+t/cTECgk5Izsh8p7l2y76e8rOKzJgtzNrDEZSjQsUJsDxd5zFb xzmtmnXHv2YZG69FlTUNI4mLoEa+2gQnYrXHRh24EZuQ9BdrLqFEl3USudU37m2FYL5Kl3xMbJ1D 2Gf39+R2ugiqNe6yGp/xBGxiix2zq8ENStOIsbpR7LO3Ln7G6UgazPnBrPyXTWzvbf4vJPf1ejiM WZL2laLhWkLC5aisENTfchkF3eb3cx1gjrMqDMMpaWBjcqwUZIDSWEaY2wtZ2GgPEmwznnWDYXGx 6N/Esm4lHnQFbQdA+rXCImZgrZgZcyNxBVRDpgI+OgwG+vjnHe/5tUGdfOWPOvFebArIADUBRWmp 4UGJwd7n7hV2tUncLeC3vgpcw2FwsI0JXc9SpkuKqJOHLVfHYhGkhEuar5/SMiuF7wwV3jLrlT4K B6A/0AX3OYiyM1p1KTkrssLN+MA1EL7h8rb9i7MNdjbxuK7QNiQ9eSAm3ZCEVX3PMdc+LAJl5x7+ xDj12N0hlAtTXAbCxvJBQR4jHsWuC81oxu5JiCegieYHHAgQAv//tES9HcBr3JDYqhKifQqQq+aw VFHMgNLspYmIQUCcN2xRnYVHf9BjpMp8JgFxbVXz2n+te38zbfXFbKRLP/kHmRjO4ZKhn/MfwR3b ApJNSPGYALIZZPGZBbxqFiM9Ji10R2pofhCTnBZWBIiYxaejPKKw3XGZcmj8YAJCmvZwVedCmS62 SoAZO8PPGgqrtmfy1NWr5BdRz7vQ0JIP41KZ1RTA54rChe6SAUrzwwBVk92EpUQkgEevr1/a4= X-QQ-XMRINFO: NI4Ajvh11aEj8Xl/2s1/T8w= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:55 +0800 X-OQ-MSGID: <20240427164159.82771-6-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 5/9] avfilter/dnn_backend_tf: Fix free context at random place X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: e4JaisBscPvE From: Zhao Zhili It will be freed again by ff_dnn_uninit. --- libavfilter/dnn/dnn_backend_tf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 60f9e57fb7..3b4de6d13f 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -804,7 +804,7 @@ err: if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { destroy_request_item(&request); } - dnn_free_model_tf(&tf_model->model); + return ret; } From patchwork Sat Apr 27 16:41:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48299 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070000pzb; Sat, 27 Apr 2024 09:42:56 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVik6QSu9TkG0CN+ICljWBE73jjWOPEOk6ZYDxqtcX3vakteC8CJ6riExl8xiLJQtqX11RFXfAvodMbZFdxviek98PWxEy+eCkF0g== X-Google-Smtp-Source: AGHT+IE7QS9EL2ex7L74+7EmdTNtoQgWf6+VMCQCOA2vd3Kb/n/nHe79cGj2Zq/DWHo8Nwj9HqCc X-Received: by 2002:a17:907:1b16:b0:a58:a0b8:32bf with SMTP id mp22-20020a1709071b1600b00a58a0b832bfmr4345665ejc.1.1714236176162; Sat, 27 Apr 2024 09:42:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236176; cv=none; d=google.com; s=arc-20160816; b=wjSmppX16R9ReEnymbOPU/BEByikQp+bdSrj/p/Mjg6QyxD2OgvVfGhs4CK9oAU5y/ CNgLlqSlgnOMpslN+Kku4IGXNALnbrg0/tdIQDcQjETZ3+3R3tGJj39Fs9n9/EZkTYhO +h5UVRUvliAQZqqqxezrP0eptTAD9Q5iZzfwQQERzREueq/cPVbXS1uKtvR2NP9Bia4a AKpY/GvKN/dB/D96YxWpHc1qqbD4GEPsKAydpA5NdW86IqfXvQbYZFmpqbgCeyTQQMw9 y7rQPFGX+jeDe1B+fgalAm19EspTDZSyVZlttw8223wKXKuHRJgvC+Eo9Al6efK1ueV5 Q6+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=9rbSvM06ZuybWkDc/57uf2VTdmLDHWjdD4rpzfZN2h0=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=jrHjDljYTmRp2ckgW08MSXMYmN4RuOdg1sW8hRGDd5fy/oAuaUPrlLDZG8sCvrPjf+ gqbcrlCiGnA1nz6wXVWK6oh0iXuXTsRgxezekOmiY19XSAeqhaNS2hqg8WyW+2bqXamN 2zsu5lroD0DglZ7ejQNrKhtSVMQzk5FixfFPhZc6QccTov/gSiEMMxLdWZwaB5fIbGMO jrnLbE1eaduXQf7vW0HyqcdiaHOehdQjVYUc74VwO5GKZrwB3Dt98+6Y3pFgwOS2P0LU 7xuixMj2T3wygZTlGQMOV0g2fpb/3B435TLaYD8efBFdh6ZPwXFw9iPSkla2vrbxqmWz cTWA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=dglaEV3k; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cw8-20020a170906478800b00a55bd0c303csi7597930ejc.62.2024.04.27.09.42.55; Sat, 27 Apr 2024 09:42:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=dglaEV3k; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 01BE368D293; Sat, 27 Apr 2024 19:42:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-82.mail.qq.com (out203-205-251-82.mail.qq.com [203.205.251.82]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B983168D220 for ; Sat, 27 Apr 2024 19:42:14 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236126; bh=4EMMI4aobqVThNJSmZXynEn2P4p5anzkIo5UKlmJIcc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=dglaEV3kWSUjiYR3R/NaBYvQ4ZW9RwCbRk7ykoY7BnLYX/+0VDsKiUJSPBGJxWyMI 1tn3R/V53j8AZrJaNfFeCuOyI2xMXzBHl74Zy0lN4r7fl7d80A/82jHGKE/0p0mfjy YQPoxLyLD50XheZM/o4KkQ21G+s9PCGeS/QE/6ZE= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236125tdeb6c7gz Message-ID: X-QQ-XMAILINFO: OQhZ3T0tjf0aNQtsHtgPWXtrTc3o8qC8fZhZrg3cT87OqUJdFWSRb9NgGW9H+X 1VhddKxCsoxKEnrIbWmxKHlsCK1TYENroTtlmIAHqiMiM9tTmAqD0jqbAL13eM6IGEzlJSyjc1Io fWy6XAwdJJFQoakGGr0vPjLii227x1ckYETrUsyQqLJM0EMca2D929Huqyu1gEF5v1r9wORQLvFg ZB5jxXwDSeu3vqsGXnRF3bg3LqskKdU33g2RBr61EhVUdemh76JjwGZfiQ2u62Bd/MzA8dHf/oLf 1guhIKtbAov1pZBB5Q24WLmeG4H6YhJ0wdhCaKFAd4TikkZ9ipKJaQe6zErVnC3ixuOpfz0sSZB8 XoGtQHPk7AhBuTONM1O5u5bHnboLTp+TbXjyg6r1CshGXgLJV9hUHZHOrKfYBGQk5MWzHti22p5L u0v4N6PhWU0wvZsJLndhmN8ceNFoJqgjPZlpncxdvsNYp8TnR5fylz+D161mKCJ9MBq3Aq1DXwmE U+GElbtRVvcPf5Du/l2tLGq1vLZiOuLPCJ18Dw2p1BKwWXfm2RQZhsQ9/PaBP8q7xS2nWhflDdCg QV+EXcf6YjAZexdef9sDCfxnV9ybgDmHJ+9rSGC5NijZVrZAKwUXdXd5F8a/SzNlevPIeZoOo2Zt ZZ3Vp2NGaOYI6iv3jJ7w4lmvdG//ZPP+flKyp2XFJn0o518S/fe8eELI5xrpVvsDX1cXHEXpzCZq b0sYTXA/bLSBUHEK3c7b7fRlSAmFFr50MRyq/dxuzSCEbEnrYIm7HKqg377a4CCpnUAIeOj6H3Uq uFUrhMH6A6i7atKRI7draxLai+csoE6efzR+fHbv6lM2BEnaCcNZEkxeDk7IJHasjc1G7i4ZZd4y h3y6Se+JaE+m/b+Q0XY1Ju9paYg+NikjA0fzGl/L8NTtdP6iooXP9GRytMZp/tZgSuqClcwekzqf PwlFPanxBNvAaLw2SIENEwg0jI5GxXPw71jmOXtst6w7kbUZ65UNJgJFbT/uN5qd6kJ+FcKu4= X-QQ-XMRINFO: M/715EihBoGSf6IYSX1iLFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:56 +0800 X-OQ-MSGID: <20240427164159.82771-7-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 6/9] avfilter/dnn_backend_tf: Simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 11qfN6I9mfyx From: Zhao Zhili --- libavfilter/dnn/dnn_backend_tf.c | 33 +++++++++++++------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 3b4de6d13f..c7716e696d 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -37,8 +37,8 @@ #include typedef struct TFModel { + DNNModel model; DnnContext *ctx; - DNNModel *model; TF_Graph *graph; TF_Session *session; TF_Status *status; @@ -518,7 +518,7 @@ static void dnn_free_model_tf(DNNModel **model) TF_DeleteStatus(tf_model->status); } av_freep(&tf_model); - av_freep(&model); + *model = NULL; } static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) @@ -526,18 +526,11 @@ static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, A DNNModel *model = NULL; TFModel *tf_model = NULL; - model = av_mallocz(sizeof(DNNModel)); - if (!model){ - return NULL; - } - tf_model = av_mallocz(sizeof(TFModel)); - if (!tf_model){ - av_freep(&model); + if (!tf_model) return NULL; - } + model = &tf_model->model; model->model = tf_model; - tf_model->model = model; tf_model->ctx = ctx; if (load_tf_model(tf_model, ctx->model_filename) != 0){ @@ -650,11 +643,11 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { } input.data = (float *)TF_TensorData(infer_request->input_tensor); - switch (tf_model->model->func_type) { + switch (tf_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (tf_model->model->frame_pre_proc != NULL) { - tf_model->model->frame_pre_proc(task->in_frame, &input, tf_model->model->filter_ctx); + if (tf_model->model.frame_pre_proc != NULL) { + tf_model->model.frame_pre_proc(task->in_frame, &input, tf_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } @@ -664,7 +657,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { ff_frame_to_dnn_detect(task->in_frame, &input, ctx); break; default: - avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model->func_type); + avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model.func_type); break; } @@ -724,12 +717,12 @@ static void infer_completion_callback(void *args) { outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); outputs[i].dt = (DNNDataType)TF_TensorType(infer_request->output_tensors[i]); } - switch (tf_model->model->func_type) { + switch (tf_model->model.func_type) { case DFT_PROCESS_FRAME: //it only support 1 output if it's frame in & frame out if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + if (tf_model->model.frame_post_proc != NULL) { + tf_model->model.frame_post_proc(task->out_frame, outputs, tf_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } @@ -741,11 +734,11 @@ static void infer_completion_callback(void *args) { } break; case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { + if (!tf_model->model.detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return; } - tf_model->model->detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + tf_model->model.detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model.filter_ctx); break; default: av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); From patchwork Sat Apr 27 16:41:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48296 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070277pzb; Sat, 27 Apr 2024 09:43:45 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVEuWpGuRerPID5t4MfSyLypHDVrwtjXmLzLJu796+sPu9RKMbyVL5Ft6JnDcyy8YhOba6ia9/ZdjbY0lllOZMTfmnwnxMc30U1AQ== X-Google-Smtp-Source: AGHT+IELvFU4lOJhIXC8xLdJPdOcBSSDZAmZY5mXgRLFCPMN+foBgC8HZ19UW1sxzD8YnNiqenvk X-Received: by 2002:a50:9541:0:b0:572:1589:eb98 with SMTP id v1-20020a509541000000b005721589eb98mr3761388eda.12.1714236225717; Sat, 27 Apr 2024 09:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236225; cv=none; d=google.com; s=arc-20160816; b=Kl+8h0meP64Hs8wKz5ZP9t9xeTJBPHFunjyAsyALR2hfoEv9ZRsYM+PGNyGiIEnJBi l2y4nUA3bEa67JzTxbHKo4uahbq8Y0nmGx1J6A8dlKJg995P2xWlXo7cCLb/nPUc3a1N k0npdi/PlHUqeaYMQPUXTYOQ46p1p0I7v+zLBNmSvrvv0br2shYyz7cAtjV+WVr8rhgf sC8cm8AuSwz4jrBBbM0MaCKVjMa0ydSuzbxl3BpLYi7IkIOttlLYJYnytBXMB/djdu+T ljKKBMFxtVRUi1J2kjg95aKSaqz/IKmReNSS0OuYs0UXX2D7oUG46UtuKnxp1R087bv1 H19g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=HjqzTmAs1Q6zrREqYzjjgudI361HRBaO0sZb1JNm/RQ=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=ea+CeqVcVuWgDdWFbju/ga8uWE6UoPMNQZCIujDgzEoL4cDqp2gRRXvYLBZe5Mh8cS C0t4m/0tmYXfwl1rEgAEmZioZpKETQUk/qXr9jjEHo9Ku593NZ39+5oOl1CZsXmvP1OJ cMvy8na8Dv9FCbCXFf40hZNIHvdmCTCsb0A96LM6cssIGXe2TjG3y9ec6JixOtyez9FQ aDoNpLfeUxDlGJJFRYj3+C3Qoo5KzoJk8aS1Q00GxMg2I9uP3DoDaXS5o7Sii3lenFS9 zY2+87GNo8zJbIwkPzgXTdvhDN2rzwm6ncuBYz03wSyXpkPxbjXGOoaNLa47lGC0Y6/l 7IGA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=I9NFg6+S; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r1-20020a50aac1000000b0056e113a5a26si11952248edc.464.2024.04.27.09.43.45; Sat, 27 Apr 2024 09:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=I9NFg6+S; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2C28F68D41A; Sat, 27 Apr 2024 19:42:31 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-240.mail.qq.com (out203-205-221-240.mail.qq.com [203.205.221.240]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2813768D2F6 for ; Sat, 27 Apr 2024 19:42:15 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236126; bh=kxUEKzo6yMwU2IjG45iacQijIia61Nk5SRdMXJQb0dw=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=I9NFg6+SlFKmWQ+mwFYAWo+YoJDF6h39bVbMJJa6NzKE6Y1jbaqNhpaHyONZgZzsb j7xCZPD53NNzWCnPMR8xn9cfLmAGKKNLFlUwPA6jk63lFkxMT6LG4eCSaE7oCYSm+y S9KJ8xQzI91jpZHZ5sk41dM50BvbF7VikGd9GNUo= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236126te19h7bm3 Message-ID: X-QQ-XMAILINFO: NHgT31LhP5vD2Jf9L9sUIZJ9hJ9D1RBsrW0L1gFHOP3vLffsjDnaGMIVliJuBU HQ6azrZ235EazlIraffJV7DukvQs8Qavk0n5al9plZlkAbGJncRezIigy7pmCBFslp7afMOxA15V LELzFRcuZ45Otw/WShqaIoNn1SyyB7deZ8/9YJD15whhs9wyeonH9oyiF7CJj+h2C1XDUFlcHLBr EPgeZ9FQMFwusMu8Xsa/11fN31PCv9L8DLwDkCNPsiRgUN7j73OK78CU7kGtAVajlDe6DIl1Vu9e qxX1ztHtd7clI3pTY3vaRL5fXyl6p6yjFmW9IJBq9CEt/tQFdQF45EzhaupOC9j/IVf4GSrI3lue 1gmh3rJJPIDDwdtqM7pGxoFjRvaw4PBgzctWDCktKZxNayaA57W9kQz4jrVuqXs+z/7siVU6O++a 8grApDJk7uuGx0YEPzDPXFhPAryqOkDCZv+jc5nYSmJGrbhP0OTVzuncl3PlT3UL0yC3HU7NkNZn IPe4gK016erNHthIoE0GjB9D+LQQXrXNCRVBU4mAMwe37EoxSvF4ftm7EZfDsbN5SfYFAdxgeeQm 4HrGI4xBs3cWsbJqlipYtA/iz5Yhs312lvJxxUqzxdEUWBiRwED9og3kIXiNIGSn1h1OY3kZkET8 yVlJ9ddO1LGKNy5+2dxD+iGLamDCvqC2/OlGl6RUobx+YFNnLqQdIaU2z4KJA8U9sqJr7vaHA5Zj PK+fkOzqBKUXgbqm1KIpprKIO+mv57+G4jrQgDR3LE7GQZEI2CCfI3++ZZfPJlEFZNYFVfHtK8jl XTA1/bdARV+4pHiVxE0yT/t/4FgFgYazpRCjWyIAo17dUuyyzn9xkXVNMH27CpkGlC/qbvhhU0PZ gLl+B+k6yGLgWDKY2WTVnP9xwXAH0cdWKAXkwj7QBYXcu4iKCHiiqhYdx4PY4+2vESyZUvGfiCdS oYIgx2lA1BLYep+y+0q0rua97ca4pQY72dVjsKzil634mxmCvDu2geEGI1POQwqk+3SPSHN4uMJC iVAZmnTA== X-QQ-XMRINFO: NS+P29fieYNw95Bth2bWPxk= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:57 +0800 X-OQ-MSGID: <20240427164159.82771-8-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 7/9] avfilter/dnn_backend_torch: Simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: zmqy/7DUi2qm From: Zhao Zhili --- libavfilter/dnn/dnn_backend_torch.cpp | 31 +++++++++++---------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index abdef1f178..818ec5b713 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -37,8 +37,8 @@ extern "C" { } typedef struct THModel { + DNNModel model; DnnContext *ctx; - DNNModel *model; torch::jit::Module *jit_model; SafeQueue *request_queue; Queue *task_queue; @@ -141,7 +141,7 @@ static void dnn_free_model_th(DNNModel **model) ff_queue_destroy(th_model->task_queue); delete th_model->jit_model; av_freep(&th_model); - av_freep(model); + *model = NULL; } static int get_input_th(void *model, DNNData *input, const char *input_name) @@ -195,19 +195,19 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) infer_request->input_tensor = new torch::Tensor(); infer_request->output = new torch::Tensor(); - switch (th_model->model->func_type) { + switch (th_model->model.func_type) { case DFT_PROCESS_FRAME: input.scale = 255; if (task->do_ioproc) { - if (th_model->model->frame_pre_proc != NULL) { - th_model->model->frame_pre_proc(task->in_frame, &input, th_model->model->filter_ctx); + if (th_model->model.frame_pre_proc != NULL) { + th_model->model.frame_pre_proc(task->in_frame, &input, th_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } } break; default: - avpriv_report_missing_feature(NULL, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(NULL, "model function type %d", th_model->model.func_type); break; } *infer_request->input_tensor = torch::from_blob(input.data, @@ -282,13 +282,13 @@ static void infer_completion_callback(void *args) { goto err; } - switch (th_model->model->func_type) { + switch (th_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { outputs.scale = 255; outputs.data = output->data_ptr(); - if (th_model->model->frame_post_proc != NULL) { - th_model->model->frame_post_proc(task->out_frame, &outputs, th_model->model->filter_ctx); + if (th_model->model.frame_post_proc != NULL) { + th_model->model.frame_post_proc(task->out_frame, &outputs, th_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, &outputs, th_model->ctx); } @@ -298,7 +298,7 @@ static void infer_completion_callback(void *args) { } break; default: - avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model.func_type); goto err; } task->inference_done++; @@ -417,17 +417,10 @@ static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, A THRequestItem *item = NULL; const char *device_name = ctx->device ? ctx->device : "cpu"; - model = (DNNModel *)av_mallocz(sizeof(DNNModel)); - if (!model) { - return NULL; - } - th_model = (THModel *)av_mallocz(sizeof(THModel)); - if (!th_model) { - av_freep(&model); + if (!th_model) return NULL; - } - th_model->model = model; + model = &th_model->model; model->model = th_model; th_model->ctx = ctx; From patchwork Sat Apr 27 16:41:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48303 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070237pzb; Sat, 27 Apr 2024 09:43:36 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUEfnmij+0szs0PC1xWKYUNWztt9FlfPwAaxuaEuu1X4TdqtjANUq8KHg+fPu5EbWQINMh0gMrNJIPh+4/umkye3bp5P5Xe+kq7RQ== X-Google-Smtp-Source: AGHT+IHnk80eoZhSYitLwoME2dyBFzvQK3MQtNniviYLyd3jEAkUb8Vf8GgQCfIPuf+3WKN3LySV X-Received: by 2002:a50:c314:0:b0:56e:966:be7c with SMTP id a20-20020a50c314000000b0056e0966be7cmr4163486edb.2.1714236216215; Sat, 27 Apr 2024 09:43:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236216; cv=none; d=google.com; s=arc-20160816; b=pyf7pN5arMXDyW2fT8L5k3/hqXTZLnoMmVc7XnI+aY42lzti6xHAkYvlh/hzB+ZqQC 4sLp3gkLKW8N8kNieADOtECl9MHJUFnju9zoTUVXaFUZCogqher6/FXXYyanxcMmJGEk N/iq1rCFdPFpXRqt/eEwN2RF4p8wmX7JnZNJaOVOUTtcR94cF6/GgdnhTobu3YkhLtwb fAF235Y4ZwMFXQXMsxoi/tb/DX5D/ZoZEfJIJhfUD6pZIv9Y/PXI8ZhdPejmYTObYLv5 NlKj+PIVIKd5k/vbTzpViMOwS/iHGvxrE88f6asGLO1Kblo+KM2Qyl8mRbh6A+Cg2uHm z4vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=IjmspWmt7bbnrn5jNYMcrLMH/RdAXugID3Lr+wYyTYI=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=TAUYvsz7fX+ircJCB8Vhrn82gKVj+ytEHbhMCvbKEb4gA4MLJN94nA9Xnb0ihSyzeZ 2PNsHDIR+2BH7Cu1IePZkxk8j7wlteajyWEw1TjBypgaa5Q+oeG3ShOUc576ZEvVbVuf 1+e3Tzxtu3XVRxyhrkSDb1HwlsrtIqCmwXp9tc/WEylciZudzhHI027CcuErcbtQqJUA 0pDOLHwYkZHr1jbihUmQljfBTXhVFla6rYiREpI+9lTytkHyVsEpTlWcNVkj5RIc3s53 HoWwGndLIv2TW0XFX1GS/BCDRkfY8wArs3NJS1osKg9HXUiaAFCaXFZwVgiL7vV9MDCz vuQw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=j62diax0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id k16-20020a056402341000b005721ebf5ef9si6087124edc.548.2024.04.27.09.43.35; Sat, 27 Apr 2024 09:43:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=j62diax0; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3151968D410; Sat, 27 Apr 2024 19:42:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-239.mail.qq.com (out203-205-221-239.mail.qq.com [203.205.221.239]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D2ACB68D256 for ; Sat, 27 Apr 2024 19:42:15 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236127; bh=37GWWd+5T8f2zM5nxJ9ueJAdGlEEv4Mg3BOMOer2xNs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=j62diax0AY6mVXw2yOTVKOhUtshW7hY3a7BEpymFdQnsFqZj5rnW2EkIVa/39EayM k2gI4mts7mv26TmGqnGwpXMsPV673QHvXTW/8rTj/ES20yDfOI/DE0pVnTNnPKKyyv MTIpRqHEI0HXEh9nqep5ii1YjyqlZt05hssf28Rk= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236126tm8je05i5 Message-ID: X-QQ-XMAILINFO: NYC8H9A5TIPOHc/ZO1CW92CcjfLTaUtEH69bVxgEtLlKrXSh1dzysABKtg9OIv Gz1VppGxvJGK+WSPNsKkYiEL9XjlGZBu4B97bNom/CxudX8KgUpZTOS9CPQZXJAp+Ni3uZvMRctj SXg7xtOmFTQ9loktJJuopq6YAhh43szOezOeZN2p5QLyizirSq1caVPabBPGqoDOtnFlNhdgJyEG R8QP58YzZFu9q6uW61X4Y43hdiUry+Bg8V0yYoyAannmyQGJ47m39UaMHwvHmebyNv+UnOWdrebS p7CXneOvanB9VRJ+kQ3+6Y03e1zqrJoHCT9eH9QRgZkjmqQjQsy9+hihu4z8M5on9F0A4nB9JARs dLtkzdUCbxDQvRaPX8QSLCle7FTC0aHnzAtmSWeW/a+gkap5ck3IgFuBh2W0gJ93PjwOEtdcRGhO BWjbm1kx3F+ihehtOcIVV5eau3/3IMaE/FG3U7cr+bpuJuR/1tVydvHwX9HVz5+mGfzvjQot5WPC jSsSyfFXu9s1aSKr2JOtrwGk/xPui+KDf+LyigBGRgEB9fqDu4R0NR4kKDILWUiDfxcX38eShhvz kcCBdnwaxlw7MZPrpgV0cyn1xmq1EhdOM4OqvKJaJhGnPHNpFIqBLU6rp0Cn/dLkuQ3oT1TRY/w1 pYW3zt6oAeL7uSTCsPsljjvt8F9EBkkR2yScAWsrlv6IQk5sx/DqfXTUDeaBxsoRTEm1OpqtlPZP HGuh0cVPNphxt3QghfZQmpGRFfpQqFN/EHMG8cKE8Ju7dzRYie52Vkx5YSxp/N3a+Z0mILwt8h7p 7Q3UkCPT/E5u95S8m22caMq5grENeEjvnM3/dkDJYEqA6rH/CX9tlbfKOAQorI+zwyzH/WuhHj/+ g1DSDnI5DfpmavsJnbbfRCqIbzrnyu4S9az5dAxEXfaqqaC0GuJ7IuGvzKq5fTR2le6bpt6Ock7j Uzy9wxJM9VW5t6RU9gE+n6OAebYnjcO57g5LNX7xaequrrySFe4D1gxQiTWKjSGqXx1/aIdnpLYh u+lro+mA== X-QQ-XMRINFO: MSVp+SPm3vtS1Vd6Y4Mggwc= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:58 +0800 X-OQ-MSGID: <20240427164159.82771-9-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 8/9] avfilter/dnn: Remove a level of dereference X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 62cC2exHkA7q From: Zhao Zhili For code such as 'model->model = ov_model' is confusing. We can just drop the member variable and use cast to get the subclass. --- libavfilter/dnn/dnn_backend_openvino.c | 17 ++++++++--------- libavfilter/dnn/dnn_backend_tf.c | 19 +++++++++---------- libavfilter/dnn/dnn_backend_torch.cpp | 15 +++++++-------- libavfilter/dnn_filter_common.c | 6 +++--- libavfilter/dnn_interface.h | 6 ++---- 5 files changed, 29 insertions(+), 34 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 1acc54b791..d8a6820dc2 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -517,7 +517,7 @@ static void dnn_free_model_ov(DNNModel **model) if (!model || !*model) return; - ov_model = (*model)->model; + ov_model = (OVModel *)(*model); while (ff_safe_queue_size(ov_model->request_queue) != 0) { OVRequestItem *item = ff_safe_queue_pop_front(ov_model->request_queue); if (item && item->infer_request) { @@ -1059,9 +1059,9 @@ err: return ret; } -static int get_input_ov(void *model, DNNData *input, const char *input_name) +static int get_input_ov(DNNModel *model, DNNData *input, const char *input_name) { - OVModel *ov_model = model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; int input_resizable = ctx->ov_option.input_resizable; @@ -1255,7 +1255,7 @@ static int extract_lltask_from_task(DNNFunctionType func_type, TaskItem *task, Q } } -static int get_output_ov(void *model, const char *input_name, int input_width, int input_height, +static int get_output_ov(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { #if HAVE_OPENVINO2 @@ -1268,7 +1268,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i input_shapes_t input_shapes; #endif int ret; - OVModel *ov_model = model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; TaskItem task; OVRequestItem *request; @@ -1383,7 +1383,6 @@ static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, A return NULL; ov_model->ctx = ctx; model = &ov_model->model; - model->model = ov_model; #if HAVE_OPENVINO2 status = ov_core_create(&core); @@ -1470,7 +1469,7 @@ err: static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_params) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; OVRequestItem *request; TaskItem *task; @@ -1558,13 +1557,13 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; return ff_dnn_get_result_common(ov_model->task_queue, in, out); } static int dnn_flush_ov(const DNNModel *model) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; OVRequestItem *request; #if HAVE_OPENVINO2 diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index c7716e696d..06ea6cbb8c 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -262,9 +262,9 @@ static TF_Tensor *allocate_input_tensor(const DNNData *input) input_dims[1] * input_dims[2] * input_dims[3] * size); } -static int get_input_tf(void *model, DNNData *input, const char *input_name) +static int get_input_tf(DNNModel *model, DNNData *input, const char *input_name) { - TFModel *tf_model = model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TF_Status *status; TF_DataType dt; @@ -310,11 +310,11 @@ static int get_input_tf(void *model, DNNData *input, const char *input_name) return 0; } -static int get_output_tf(void *model, const char *input_name, int input_width, int input_height, +static int get_output_tf(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { int ret; - TFModel *tf_model = model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TaskItem task; TFRequestItem *request; @@ -486,7 +486,7 @@ static void dnn_free_model_tf(DNNModel **model) if (!model || !*model) return; - tf_model = (*model)->model; + tf_model = (TFModel *)(*model); while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); destroy_request_item(&item); @@ -530,7 +530,6 @@ static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, A if (!tf_model) return NULL; model = &tf_model->model; - model->model = tf_model; tf_model->ctx = ctx; if (load_tf_model(tf_model, ctx->model_filename) != 0){ @@ -611,7 +610,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { task = lltask->task; request->lltask = lltask; - ret = get_input_tf(tf_model, &input, task->input_name); + ret = get_input_tf(&tf_model->model, &input, task->input_name); if (ret != 0) { goto err; } @@ -803,7 +802,7 @@ err: static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TaskItem *task; TFRequestItem *request; @@ -851,13 +850,13 @@ static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; return ff_dnn_get_result_common(tf_model->task_queue, in, out); } static int dnn_flush_tf(const DNNModel *model) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TFRequestItem *request; int ret; diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 818ec5b713..24e9f2c8e2 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -119,7 +119,7 @@ static void dnn_free_model_th(DNNModel **model) if (!model || !*model) return; - th_model = (THModel *) (*model)->model; + th_model = (THModel *) (*model); while (ff_safe_queue_size(th_model->request_queue) != 0) { THRequestItem *item = (THRequestItem *)ff_safe_queue_pop_front(th_model->request_queue); destroy_request_item(&item); @@ -144,7 +144,7 @@ static void dnn_free_model_th(DNNModel **model) *model = NULL; } -static int get_input_th(void *model, DNNData *input, const char *input_name) +static int get_input_th(DNNModel *model, DNNData *input, const char *input_name) { input->dt = DNN_FLOAT; input->order = DCO_RGB; @@ -179,7 +179,7 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) task = lltask->task; infer_request = request->infer_request; - ret = get_input_th(th_model, &input, NULL); + ret = get_input_th(&th_model->model, &input, NULL); if ( ret != 0) { goto err; } @@ -356,7 +356,7 @@ err: return ret; } -static int get_output_th(void *model, const char *input_name, int input_width, int input_height, +static int get_output_th(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { int ret = 0; @@ -421,7 +421,6 @@ static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, A if (!th_model) return NULL; model = &th_model->model; - model->model = th_model; th_model->ctx = ctx; c10::Device device = c10::Device(device_name); @@ -489,7 +488,7 @@ fail: static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_params) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; DnnContext *ctx = th_model->ctx; TaskItem *task; THRequestItem *request; @@ -538,13 +537,13 @@ static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame **in, AVFrame **out) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; return ff_dnn_get_result_common(th_model->task_queue, in, out); } static int dnn_flush_th(const DNNModel *model) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; THRequestItem *request; if (ff_queue_size(th_model->lltask_queue) == 0) diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 3dd51badf6..132dd75550 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -151,15 +151,15 @@ int ff_dnn_set_classify_post_proc(DnnContext *ctx, ClassifyPostProc post_proc) int ff_dnn_get_input(DnnContext *ctx, DNNData *input) { - return ctx->model->get_input(ctx->model->model, input, ctx->model_inputname); + return ctx->model->get_input(ctx->model, input, ctx->model_inputname); } int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height) { char * output_name = ctx->model_outputnames && ctx->backend_type != DNN_TH ? ctx->model_outputnames[0] : NULL; - return ctx->model->get_output(ctx->model->model, ctx->model_inputname, input_width, input_height, - (const char *)output_name, output_width, output_height); + return ctx->model->get_output(ctx->model, ctx->model_inputname, input_width, input_height, + (const char *)output_name, output_width, output_height); } int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame) diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 9516130714..2b35c9c9fb 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -91,17 +91,15 @@ typedef int (*DetectPostProc)(AVFrame *frame, DNNData *output, uint32_t nb, AVFi typedef int (*ClassifyPostProc)(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx); typedef struct DNNModel{ - // Stores model that can be different for different backends. - void *model; // Stores FilterContext used for the interaction between AVFrame and DNNData AVFilterContext *filter_ctx; // Stores function type of the model DNNFunctionType func_type; // Gets model input information // Just reuse struct DNNData here, actually the DNNData.data field is not needed. - int (*get_input)(void *model, DNNData *input, const char *input_name); + int (*get_input)(struct DNNModel *model, DNNData *input, const char *input_name); // Gets model output width/height with given input w/h - int (*get_output)(void *model, const char *input_name, int input_width, int input_height, + int (*get_output)(struct DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height); // set the pre process to transfer data from AVFrame to DNNData // the default implementation within DNN is used if it is not provided by the filter From patchwork Sat Apr 27 16:41:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48304 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:1509:b0:1a9:af23:56c1 with SMTP id nq9csp1070352pzb; Sat, 27 Apr 2024 09:43:57 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUi4uVS8+JofgzywAHXOQUDIIvy+WmtQ82HG7u5cS4+KXcc/CiAWak1Axg5PO+K2q8fwlyPgQ3tDjGGfYyMdsiecUKIv/KnYRxvLg== X-Google-Smtp-Source: AGHT+IFUaOtZnawYOxZxoywVkjX61F4g36Do8cbQp/Jhz2LrmnsPYPgi0Ykngo0mFFY49GpTsQEW X-Received: by 2002:a17:907:720c:b0:a58:e59c:58e2 with SMTP id dr12-20020a170907720c00b00a58e59c58e2mr1494673ejc.7.1714236237074; Sat, 27 Apr 2024 09:43:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714236237; cv=none; d=google.com; s=arc-20160816; b=nhGbso0x1bbEosp66f2At99/lkuefYKXklqY3JO9qQvIwnKa8a3jarPPBY938eddlj f7QW1dPoHEL0rX0VSq4lnHPCQBNgJ+pm1VN8PO8LX9fZ2+QkOsCzoCUsVPHsDPwOYq9k NnOoQQumKmxkdhOT6VSI8n/Uqh8Elisr0t9UClmWFwquu6AvzcYb23vPgMoxyvOlD3zp jT1WwGQ+WPXq9hXUW3EsVJ4ukrxbIqKbx8q6mMQhUHMBtbU/BGZESqj/ewq6Ekk088IP XM9YJkfjOrcZ0hue+z7ovez4wBPUPEtGLiSxrLBZlHzPBbaBHKkKZEr13n4xMzuTbaOZ lTpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=M8YMeIfFnepOGXTv09nEVY8n1Ob+DMX9wJ/3TSiSbp4=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=CK+RVj55QwwWOnvHhM5fFO/1G1XMzCp0rg2y2hbm18l/gaj3/kOIg1bTJwE3sY2OOb 03/nt+IJ/zv3rjmyEmGbefVV9IpF/S4QAuAv/YrKtn7r4Ri1FEca3DgGEnFB5R8z7tI4 OazTJtRpZD07xIfGgLRzci58sT6zqm0gE5u3U8lsPzt8Gd6E4mRQcrvueVKQsmeFq4+D TtzrTjuYzxgUMCi9wTcBeRFQ4x9xpSladv1I0kwwbrE3KlQhU5H3SwtIh3uF3mCCKJzx 5TfiSWlwcJx2xmWXBKldbED1+F2VHAGaqRlTFIRMj3wT21YNU/l/tz2o/dZJkjwJIX0/ sPyQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=JjvuLMnW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id qf39-20020a1709077f2700b00a55b9c76da4si8344803ejc.761.2024.04.27.09.43.54; Sat, 27 Apr 2024 09:43:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=JjvuLMnW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6118A68D427; Sat, 27 Apr 2024 19:42:32 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-235.mail.qq.com (out203-205-221-235.mail.qq.com [203.205.221.235]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 426D968D33D for ; Sat, 27 Apr 2024 19:42:16 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714236128; bh=WytxIeKHgoeS51VRjPoS7RvFX77IU+L+e55zh+/4Fwo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JjvuLMnW/6lGP7b/G1lt3cWlYc2sRFTysFQdoVp2mAXeYJcRR7K0eEvidRKsp8CMG WAzXUAxVfDAYih33yIMIJtPrSMHnis4IPe36nuCPBzhK0o4+4ql6WSKFyqeZ73q4dK fBGQT4hGF53n3laP8f0gTjaY0ESvHr4oml/3QjWE= Received: from localhost.localdomain ([113.118.107.199]) by newxmesmtplogicsvrszc19-0.qq.com (NewEsmtp) with SMTP id A8200E2A; Sun, 28 Apr 2024 00:42:02 +0800 X-QQ-mid: xmsmtpt1714236127t0o114ehs Message-ID: X-QQ-XMAILINFO: NYqi3QrBgYD/ruq+g1+7tr/h5wrWSzU0cotHf+t6NYXZkOsYEm8vuBBPCxQnD6 teltBbjP76PhhNRDKerflLUQyo2DMJESP8HXyg0wZ7Y9YVG5hXRiGOX1+lENYGWbtSeScUHnBjky WP/GFLcZuBH8U2qao0eAP1GG8ZDNUB/tgfCvzheHNMBDXXX7CKI+BIMbKYQk+7wh6RhIhbsoQkSj /FFL7GRweYLZ7iuVp4YET9PS4CpswhSb7xl8BDzAd8/EeJ4oYlWVxceBWms3fzPjx0Xe9UoaxdEa 26zfY5JT5ICiOLWiHUpwY6ohIf0NqSbTz0AsjDEXt+gDkA88ciFZxXm+LZkPQydeXxI99vAbSRqD 4JCPKddEk3St/Ppf9UPwNg9r8oxKc5jLgTjKHeo4qTNd6qBGkMnEpjW0vC7u/+mQkOrR73ZN3f0q 3cZ2jnKU1+j/1j4c0P26parJK4pYPL3OHRlSrrlxVwsEP4xJ/7ERvHkOBoxt43+kaCFft0V0OmFn mjGDm8AEeDh0HZiBjPo/WOFhRJUX31Sin7BOxNfCFSCH9HmJ67hxOi8uoB3kDFW/rxL+yDuqAXpl 9zo4ZPtxqcTyxB+P5G8OZXlfL8OC73AZnyW+QYCNd0Ic5w6Diy8fhrh2QN33CIxl1254WPWMDins QAayvU+0Jq8/mufR61eAW1b3lsjm2aP59fi165DIwT2J+TKglfgWjYXF27+pDsreEuO1vbdD/NVr prR/Izm8hxUbVccHxBZA7GpH4owTLn9df03gisDvHUWn0ZxpkGxVkQLawmIU3yPX9Rzp+D2M0HD7 CXbSpklcDnQ9JLoBqQQri6B41D/5TeNOHARZAs3F2LJ+NSjUSMADAkvW9+EuDanjia+kxJ+h8fN/ 9C3vEBJjZ9ZRRx3xxl86orRzJaXTuN10QkzmczMneEyuosAKiKMHgRoanBz3SsP5ec0z4TNpq+wX S5yG8zsD2YJ08cD+QcXwhMsQb/rSeTm3Oz9Vht7Sg= X-QQ-XMRINFO: OD9hHCdaPRBwq3WW+NvGbIU= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 28 Apr 2024 00:41:59 +0800 X-OQ-MSGID: <20240427164159.82771-10-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240427164159.82771-1-quinkblack@foxmail.com> References: <20240427164159.82771-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH WIP 9/9] avfilter/dnn: Use dnn_backend_info_list to search for dnn module X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 8oNR3uwwVA4X From: Zhao Zhili --- libavfilter/dnn/dnn_backend_openvino.c | 1 + libavfilter/dnn/dnn_backend_tf.c | 1 + libavfilter/dnn/dnn_backend_torch.cpp | 1 + libavfilter/dnn/dnn_interface.c | 26 ++++++++------------------ libavfilter/dnn_interface.h | 1 + 5 files changed, 12 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index d8a6820dc2..9c699cdc8c 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -1613,6 +1613,7 @@ static int dnn_flush_ov(const DNNModel *model) const DNNModule ff_dnn_backend_openvino = { .clazz = DNN_DEFINE_CLASS(dnn_openvino), + .type = DNN_OV, .load_model = dnn_load_model_ov, .execute_model = dnn_execute_model_ov, .get_result = dnn_get_result_ov, diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 06ea6cbb8c..6afefe8115 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -886,6 +886,7 @@ static int dnn_flush_tf(const DNNModel *model) const DNNModule ff_dnn_backend_tf = { .clazz = DNN_DEFINE_CLASS(dnn_tensorflow), + .type = DNN_TF, .load_model = dnn_load_model_tf, .execute_model = dnn_execute_model_tf, .get_result = dnn_get_result_tf, diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 24e9f2c8e2..2557264713 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -561,6 +561,7 @@ static int dnn_flush_th(const DNNModel *model) extern const DNNModule ff_dnn_backend_torch = { .clazz = DNN_DEFINE_CLASS(dnn_th), + .type = DNN_TH, .load_model = dnn_load_model_th, .execute_model = dnn_execute_model_th, .get_result = dnn_get_result_th, diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 245fe5f759..76fe364a47 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -80,25 +80,15 @@ static const DnnBackendInfo dnn_backend_info_list[] = { const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) { - switch(backend_type){ - #if (CONFIG_LIBTENSORFLOW == 1) - case DNN_TF: - return &ff_dnn_backend_tf; - #endif - #if (CONFIG_LIBOPENVINO == 1) - case DNN_OV: - return &ff_dnn_backend_openvino; - #endif - #if (CONFIG_LIBTORCH == 1) - case DNN_TH: - return &ff_dnn_backend_torch; - #endif - default: - av_log(log_ctx, AV_LOG_ERROR, - "Module backend_type %d is not supported or enabled.\n", - backend_type); - return NULL; + for (int i = 1; i < FF_ARRAY_ELEMS(dnn_backend_info_list); i++) { + if (dnn_backend_info_list[i].module->type == backend_type) + return dnn_backend_info_list[i].module; } + + av_log(log_ctx, AV_LOG_ERROR, + "Module backend_type %d is not supported or enabled.\n", + backend_type); + return NULL; } void *ff_dnn_child_next(DnnContext *obj, void *prev) { diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 2b35c9c9fb..3fe88d4029 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -170,6 +170,7 @@ typedef struct DnnContext { // Stores pointers to functions for loading, executing, freeing DNN models for one of the backends. struct DNNModule { const AVClass clazz; + DNNBackendType type; // Loads model and parameters from given file. Returns NULL if it is not possible. DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); // Executes model with specified input and output. Returns the error code otherwise.