From patchwork Tue Apr 30 07:11:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48426 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323127pza; Tue, 30 Apr 2024 00:13:28 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCX8cDj4N9DY5k4uoGylIlOD6IWaFTnByDDPHRMm2wzn1G/XmPAsZS3QFJ8aRNi7pwb+9K+2AAhXA7kuJ1J4XYorPlElJ1zfMUnE2w== X-Google-Smtp-Source: AGHT+IHmh7CeBWvM83QbEO6UtpXGOcw0gGH8ZphHP252GvvZAyADc/jhiz3ZMl74yIC6l5UqVmWK X-Received: by 2002:a19:2d44:0:b0:51a:d7b7:7d7 with SMTP id t4-20020a192d44000000b0051ad7b707d7mr7714910lft.3.1714461208646; Tue, 30 Apr 2024 00:13:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461208; cv=none; d=google.com; s=arc-20160816; b=HAvH8eBgIXpl4HAtSrpCegTeGut8W1lmE9Uo1SH7gRUx2Xe1f+KlGIhQ86EGeq50Uo 0XuyWjvDu+3OcUDftU8NRo/19l0EoXcVxd4xkPaH+QE3RWxd/dn+gVCgWaaPLeLxRr3U XsFT70qmBgQre9zqic1Z2IKvOFiHcM/lTidqTd2TWuoyqLxE5T4krDPFgCl7XW1skzTg ur9p5Ks+Ye/eJrgMpiGuiqiqn/U4GhVfyqdKY6fti/CiNonf6i8URxllvyi4IyUEouv/ J3hz1Zq0gtIp+JcVwaE7nCV5erynfqu+jJ0wIRrhM4KVzVa938CSONDiRtrbVCjG92XK TPpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=E//WJG6zidt4Y+8e0sfzLJ7WecmnIUqQ/GHnmyhgdDg=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=twgefGkEwvaUmYsWHQwRvK3aJ8XVdue4OvCXyJwj79k5CHjkIkU6ritiAdUr2c1VlM 6KFQAv6H08b/bmKLzVGkoJCNO7XVrjhv7Il1FEoV0wMn8RRBahVjH/hvPN2Ag02Fs0PE X2c+0FKG+501oI6Y+PaRDaR0f3WJPKsbsrxlxL1uH2hVRZOhfRIBzSUtYLeZgq5zA5vx wm6lG01erkyvmj5tc5w7gsuXHA6hZ9PZfpNKdRpr47+RWAo50M8KQ1N43VIrN5xKCsyb q4eoQfdBC1mGURJ+GBOQcHBx11jk/nzqZF3S/s+t1vNjduRkCKjFRsrIsEz5Zf84vwc5 5xgg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=wfVvRIij; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x5-20020a056512130500b005131b424439si6951466lfu.252.2024.04.30.00.13.28; Tue, 30 Apr 2024 00:13:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=wfVvRIij; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 14B8768D5E8; Tue, 30 Apr 2024 10:12:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-49.mail.qq.com (out162-62-57-49.mail.qq.com [162.62.57.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4D77868D5A3 for ; Tue, 30 Apr 2024 10:12:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461133; bh=C9FRTSfC/sJnZqPGwidzCG6HGZD53Q9RVA+LkB1Ifh0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=wfVvRIijmJFCOKHIns8+jnPXUvk+EfBHbh2N45Xx/47aCbWEtpkKNszG//wfQfDZ9 lN7TQ1obSljPpjCU8xDChCvILFiIu3nrq+bdfykc2c3UFs+9/ctT3HNntxWKs3Dsnf bO8+LEDQUfL+/JKQABBU4MXQu/fEPlGFQFLyqOIM= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461130to4c7g8ps Message-ID: X-QQ-XMAILINFO: MmpliBmRb3iCcN1WUoFrt4LP6SBB2MwLWrtKvAPr2YfSG1nlk/XjdEwTL1VrQq gUghfGS91w2wRCuQ7ReQBxIqBUAKbOSHcXdkagjaqvaPpuVuGNcYylKoGidM+UkCwMxEP4rvhI2h n/QMvlXzPsCFhmKPuztBNalJ3DtJpti396Zx0JNWzLGXDYBwJSQW6G1lxrryu4KDmJQNmxUxMuRn hfG/a9lN1K2KsFag6Zjt6IVEpNU72ka/NYtirKf+lzb/26n0VMxn4hFWAl3Jqetyh631wn9nVP59 R1s0fcDHKnKiTvLxI37sChpiCFq1U17llviOup0Kushc7BUq9U0cFuOlMGoqinwwcctBFbsboyn0 LkNu/afM8ectxdnchUyM1T4159yp7hJcV/F1a62b8ctDxNzFa76fZYBu8KeegTzcFP5HoafNcxAx R/0dWdu2SwheSp+gnDzD9DvVM3kKL8tMGSXZHDfxJr8IbHJ0sedjntqki9HStSB4DZtns/SxKQ+1 7cfCgJ/RKr8kPjPh6AS/f0Po1lDZNs2m3/V7CUmYp3EVAuMsfZiR18N7NMWj+l5qUk2Cn/fpbafo NHs8qcKDlrhFXtJa4tUsHCpypVF++0dTwvnisYe5tVX9fQMZgC1QFCI0tVgvSh+vMmdhuj55vyFF BVsfsHXvqwlkFMN8i3eCn9pzRYOZc87mOJJ2uDG/8t3Uka1q+TDDf3mykEVPrzGa9095Nqj8P1Sv k1Iw0H05BasJJlXhSxDVbt/mhr8/PeoqeC+Btow+yoOqt4UR1XpiJldSEWtjup/4G2UrH7aoTnBV kyoK7mPZzdIsuq+N4uIdGWQ1D9tQ3thTxRJLoBFoe/wEitSiHrXghj1Pco/OD9RvdI/rej7I3cz8 PbMcwa6HNCMlVjyAcCMH+0bUaT76Xi9zx18WYDsn3QvwnJ14Iv5DbCgggz9Detkz0gYKjjtg11At lwT8izosSvTgvtBqhiOz7wWjIwPccotmN875byM9vQGWorAxa8tEzLR2HzzO61 X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:11:59 +0800 X-OQ-MSGID: <20240430071208.126817-2-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 01/10] avfilter/dnn: Refactor DNN parameter configuration system X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZEdQoovOn04U From: Zhao Zhili This patch trying to resolve mulitiple issues related to parameter configuration: Firstly, each DNN filters duplicate DNN_COMMON_OPTIONS, which should be the common options of backend. Secondly, backend options are hidden behind the scene. It's a AV_OPT_TYPE_STRING backend_configs for user, and parsed by each backend. We don't know each backend support what kind of options from the help message. Third, DNN backends duplicate DNN_BACKEND_COMMON_OPTIONS. Last but not the least, pass backend options via AV_OPT_TYPE_STRING makes it hard to pass AV_OPT_TYPE_BINARY to backend, if not impossible. This patch puts backend common options and each backend options inside DnnContext to reduce code duplication, make options user friendly, and easy to extend for future usecase. There is a known issue that, for a filter which only support one or two of the backends, the help message still show the option of all three backends. Each DNN filter should be able to run on any backend. Current issue is mostly due to incomplete implementation (e.g., libtorch only support DFT_PROCESS_FRAME), and lack of maintenance on the filters. For example, ./ffmpeg -h filter=dnn_processing dnn_processing AVOptions: dnn_backend ..FV....... DNN backend (from INT_MIN to INT_MAX) (default tensorflow) tensorflow 1 ..FV....... tensorflow backend flag openvino 2 ..FV....... openvino backend flag torch 3 ..FV....... torch backend flag dnn_base AVOptions: model ..F........ path to model file input ..F........ input name of the model output ..F........ output name of the model backend_configs ..F.......P backend configs (deprecated) options ..F.......P backend configs (deprecated) nireq ..F........ number of request (from 0 to INT_MAX) (default 0) async ..F........ use DNN async inference (default true) device ..F........ device to run model dnn_tensorflow AVOptions: sess_config ..F........ config for SessionOptions dnn_openvino AVOptions: batch_size ..F........ batch size per request (from 1 to 1000) (default 1) input_resizable ..F........ can input be resizable or not (default false) layout ..F........ input layout of model (from 0 to 2) (default none) none 0 ..F........ none nchw 1 ..F........ nchw nhwc 2 ..F........ nhwc scale ..F........ Add scale preprocess operation. Divide each element of input by specified value. (from INT_MIN to INT_MAX) (default 0) mean ..F........ Add mean preprocess operation. Subtract specified value from each element of input. (from INT_MIN to INT_MAX) (default 0) dnn_th AVOptions: optimize ..F........ turn on graph executor optimization (from 0 to 1) (default 0) --- libavfilter/dnn/dnn_backend_common.h | 13 ++- libavfilter/dnn/dnn_backend_openvino.c | 146 ++++++++++--------------- libavfilter/dnn/dnn_backend_tf.c | 82 +++++--------- libavfilter/dnn/dnn_backend_torch.cpp | 67 ++++-------- libavfilter/dnn/dnn_interface.c | 89 +++++++++++++++ libavfilter/dnn_filter_common.c | 38 ++++++- libavfilter/dnn_filter_common.h | 39 +++---- libavfilter/dnn_interface.h | 67 +++++++++++- libavfilter/vf_derain.c | 6 +- libavfilter/vf_dnn_classify.c | 4 +- libavfilter/vf_dnn_detect.c | 4 +- libavfilter/vf_dnn_processing.c | 4 +- libavfilter/vf_sr.c | 6 +- 13 files changed, 336 insertions(+), 229 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 42c67c7040..9f5d37b3e0 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -28,9 +28,16 @@ #include "../dnn_interface.h" #include "libavutil/thread.h" -#define DNN_BACKEND_COMMON_OPTIONS \ - { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, \ - { "async", "use DNN async inference", OFFSET(options.async), AV_OPT_TYPE_BOOL, { .i64 = 1 }, 0, 1, FLAGS }, +#define DNN_DEFINE_CLASS_EXT(name, desc, options) \ + { \ + .class_name = desc, \ + .item_name = av_default_item_name, \ + .option = options, \ + .version = LIBAVUTIL_VERSION_INT, \ + .category = AV_CLASS_CATEGORY_FILTER, \ + } +#define DNN_DEFINE_CLASS(fname) \ + DNN_DEFINE_CLASS_EXT(fname, #fname, fname##_options) // one task for one function call from dnn interface typedef struct TaskItem { diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 374f21b7a1..c4b0682f11 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -40,24 +40,8 @@ #endif #include "dnn_backend_common.h" -typedef struct OVOptions{ - char *device_type; - int nireq; - uint8_t async; - int batch_size; - int input_resizable; - DNNLayout layout; - float scale; - float mean; -} OVOptions; - -typedef struct OVContext { - const AVClass *class; - OVOptions options; -} OVContext; - typedef struct OVModel{ - OVContext ctx; + DnnContext *ctx; DNNModel *model; #if HAVE_OPENVINO2 ov_core_t *core; @@ -98,24 +82,20 @@ typedef struct OVRequestItem { generated_string = generated_string ? av_asprintf("%s %s", generated_string, iterate_string) : \ av_asprintf("%s", iterate_string); -#define OFFSET(x) offsetof(OVContext, x) +#define OFFSET(x) offsetof(OVOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_openvino_options[] = { - { "device", "device to run model", OFFSET(options.device_type), AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS }, - DNN_BACKEND_COMMON_OPTIONS - { "batch_size", "batch size per request", OFFSET(options.batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, - { "input_resizable", "can input be resizable or not", OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, - { "layout", "input layout of model", OFFSET(options.layout), AV_OPT_TYPE_INT, { .i64 = DL_NONE}, DL_NONE, DL_NHWC, FLAGS, .unit = "layout" }, + { "batch_size", "batch size per request", OFFSET(batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, + { "input_resizable", "can input be resizable or not", OFFSET(input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "layout", "input layout of model", OFFSET(layout), AV_OPT_TYPE_INT, { .i64 = DL_NONE}, DL_NONE, DL_NHWC, FLAGS, .unit = "layout" }, { "none", "none", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NONE }, 0, 0, FLAGS, .unit = "layout"}, { "nchw", "nchw", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NCHW }, 0, 0, FLAGS, .unit = "layout"}, { "nhwc", "nhwc", 0, AV_OPT_TYPE_CONST, { .i64 = DL_NHWC }, 0, 0, FLAGS, .unit = "layout"}, - { "scale", "Add scale preprocess operation. Divide each element of input by specified value.", OFFSET(options.scale), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, - { "mean", "Add mean preprocess operation. Subtract specified value from each element of input.", OFFSET(options.mean), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, + { "scale", "Add scale preprocess operation. Divide each element of input by specified value.", OFFSET(scale), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, + { "mean", "Add mean preprocess operation. Subtract specified value from each element of input.", OFFSET(mean), AV_OPT_TYPE_FLOAT, { .dbl = 0 }, INT_MIN, INT_MAX, FLAGS}, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_openvino); - #if HAVE_OPENVINO2 static const struct { ov_status_e status; @@ -199,7 +179,7 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) DNNData input; LastLevelTaskItem *lltask; TaskItem *task; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 int64_t* dims; ov_status_e status; @@ -292,7 +272,7 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) input.scale = 1; input.mean = 0; - for (int i = 0; i < ctx->options.batch_size; ++i) { + for (int i = 0; i < ctx->ov_option.batch_size; ++i) { lltask = ff_queue_pop_front(ov_model->lltask_queue); if (!lltask) { break; @@ -360,7 +340,7 @@ static void infer_completion_callback(void *args) OVModel *ov_model = task->model; SafeQueue *requestq = ov_model->request_queue; DNNData *outputs; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 size_t* dims; ov_status_e status; @@ -410,9 +390,9 @@ static void infer_completion_callback(void *args) outputs[i].dims[2] = output_shape.rank > 1 ? dims[output_shape.rank - 2] : 1; outputs[i].dims[3] = output_shape.rank > 0 ? dims[output_shape.rank - 1] : 1; av_assert0(request->lltask_count <= dims[0]); - outputs[i].layout = ctx->options.layout; - outputs[i].scale = ctx->options.scale; - outputs[i].mean = ctx->options.mean; + outputs[i].layout = ctx->ov_option.layout; + outputs[i].scale = ctx->ov_option.scale; + outputs[i].mean = ctx->ov_option.mean; ov_shape_free(&output_shape); ov_tensor_free(output_tensor); output_tensor = NULL; @@ -452,9 +432,9 @@ static void infer_completion_callback(void *args) output.dims[i] = dims.dims[i]; av_assert0(request->lltask_count <= dims.dims[0]); output.dt = precision_to_datatype(precision); - output.layout = ctx->options.layout; - output.scale = ctx->options.scale; - output.mean = ctx->options.mean; + output.layout = ctx->ov_option.layout; + output.scale = ctx->ov_option.scale; + output.mean = ctx->ov_option.mean; outputs = &output; #endif @@ -590,7 +570,6 @@ static void dnn_free_model_ov(DNNModel **model) av_free(ov_model->all_output_names); av_free(ov_model->all_input_names); #endif - av_opt_free(&ov_model->ctx); av_freep(&ov_model); av_freep(model); } @@ -599,7 +578,7 @@ static void dnn_free_model_ov(DNNModel **model) static int init_model_ov(OVModel *ov_model, const char *input_name, const char **output_names, int nb_outputs) { int ret = 0; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 ov_status_e status; ov_preprocess_input_tensor_info_t* input_tensor_info = NULL; @@ -610,7 +589,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_layout_t* NCHW_layout = NULL; const char* NHWC_desc = "NHWC"; const char* NCHW_desc = "NCHW"; - const char* device = ctx->options.device_type; + const char* device = ctx->device ? ctx->device : "CPU"; #else IEStatusCode status; ie_available_devices_t a_dev; @@ -618,17 +597,17 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * char *all_dev_names = NULL; #endif // We scale pixel by default when do frame processing. - if (fabsf(ctx->options.scale) < 1e-6f) - ctx->options.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; + if (fabsf(ctx->ov_option.scale) < 1e-6f) + ctx->ov_option.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; // batch size - if (ctx->options.batch_size <= 0) { - ctx->options.batch_size = 1; + if (ctx->ov_option.batch_size <= 0) { + ctx->ov_option.batch_size = 1; } #if HAVE_OPENVINO2 - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { avpriv_report_missing_feature(ctx, "Do not support batch_size > 1 for now," "change batch_size to 1.\n"); - ctx->options.batch_size = 1; + ctx->ov_option.batch_size = 1; } status = ov_preprocess_prepostprocessor_create(ov_model->ov_model, &ov_model->preprocess); @@ -677,9 +656,9 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ret = ov2_map_error(status, NULL); goto err; } - if (ctx->options.layout == DL_NCHW) + if (ctx->ov_option.layout == DL_NCHW) status = ov_preprocess_input_model_info_set_layout(input_model_info, NCHW_layout); - else if (ctx->options.layout == DL_NHWC) + else if (ctx->ov_option.layout == DL_NHWC) status = ov_preprocess_input_model_info_set_layout(input_model_info, NHWC_layout); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to get set input model layout\n"); @@ -725,7 +704,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } if (ov_model->model->func_type != DFT_PROCESS_FRAME) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); - else if (fabsf(ctx->options.scale - 1) > 1e-6f || fabsf(ctx->options.mean) > 1e-6f) + else if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); else status |= ov_preprocess_output_set_element_type(output_tensor_info, U8); @@ -740,7 +719,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_model->output_info = NULL; } // set preprocess steps. - if (fabsf(ctx->options.scale - 1) > 1e-6f || fabsf(ctx->options.mean) > 1e-6f) { + if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) { ov_preprocess_preprocess_steps_t* input_process_steps = NULL; status = ov_preprocess_input_info_get_preprocess_steps(ov_model->input_info, &input_process_steps); if (status != OK) { @@ -749,8 +728,8 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } status = ov_preprocess_preprocess_steps_convert_element_type(input_process_steps, F32); - status |= ov_preprocess_preprocess_steps_mean(input_process_steps, ctx->options.mean); - status |= ov_preprocess_preprocess_steps_scale(input_process_steps, ctx->options.scale); + status |= ov_preprocess_preprocess_steps_mean(input_process_steps, ctx->ov_option.mean); + status |= ov_preprocess_preprocess_steps_scale(input_process_steps, ctx->ov_option.scale); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to set preprocess steps\n"); ov_preprocess_preprocess_steps_free(input_process_steps); @@ -824,7 +803,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ov_layout_free(NCHW_layout); ov_layout_free(NHWC_layout); #else - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { input_shapes_t input_shapes; status = ie_network_get_input_shapes(ov_model->network, &input_shapes); if (status != OK) { @@ -832,7 +811,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } for (int i = 0; i < input_shapes.shape_num; i++) - input_shapes.shapes[i].shape.dims[0] = ctx->options.batch_size; + input_shapes.shapes[i].shape.dims[0] = ctx->ov_option.batch_size; status = ie_network_reshape(ov_model->network, input_shapes); ie_network_input_shapes_free(&input_shapes); if (status != OK) { @@ -882,7 +861,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } } - status = ie_core_load_network(ov_model->core, ov_model->network, ctx->options.device_type, &config, &ov_model->exe_network); + status = ie_core_load_network(ov_model->core, ov_model->network, ctx->device, &config, &ov_model->exe_network); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to load OpenVINO model network\n"); status = ie_core_get_available_devices(ov_model->core, &a_dev); @@ -895,15 +874,15 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * APPEND_STRING(all_dev_names, a_dev.devices[i]) } av_log(ctx, AV_LOG_ERROR,"device %s may not be supported, all available devices are: \"%s\"\n", - ctx->options.device_type, all_dev_names); + ctx->device, all_dev_names); ret = AVERROR(ENODEV); goto err; } #endif // create infer_requests for async execution - if (ctx->options.nireq <= 0) { + if (ctx->nireq <= 0) { // the default value is a rough estimation - ctx->options.nireq = av_cpu_count() / 2 + 1; + ctx->nireq = av_cpu_count() / 2 + 1; } ov_model->request_queue = ff_safe_queue_create(); @@ -912,7 +891,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * goto err; } - for (int i = 0; i < ctx->options.nireq; i++) { + for (int i = 0; i < ctx->nireq; i++) { OVRequestItem *item = av_mallocz(sizeof(*item)); if (!item) { ret = AVERROR(ENOMEM); @@ -945,7 +924,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * } #endif - item->lltasks = av_malloc_array(ctx->options.batch_size, sizeof(*item->lltasks)); + item->lltasks = av_malloc_array(ctx->ov_option.batch_size, sizeof(*item->lltasks)); if (!item->lltasks) { ret = AVERROR(ENOMEM); goto err; @@ -994,7 +973,7 @@ static int execute_model_ov(OVRequestItem *request, Queue *inferenceq) LastLevelTaskItem *lltask; int ret = 0; TaskItem *task; - OVContext *ctx; + DnnContext *ctx; OVModel *ov_model; if (ff_queue_size(inferenceq) == 0) { @@ -1010,7 +989,7 @@ static int execute_model_ov(OVRequestItem *request, Queue *inferenceq) lltask = ff_queue_peek_front(inferenceq); task = lltask->task; ov_model = task->model; - ctx = &ov_model->ctx; + ctx = ov_model->ctx; ret = fill_model_input_ov(ov_model, request); if (ret != 0) { @@ -1084,8 +1063,8 @@ err: static int get_input_ov(void *model, DNNData *input, const char *input_name) { OVModel *ov_model = model; - OVContext *ctx = &ov_model->ctx; - int input_resizable = ctx->options.input_resizable; + DnnContext *ctx = ov_model->ctx; + int input_resizable = ctx->ov_option.input_resizable; #if HAVE_OPENVINO2 ov_shape_t input_shape = {0}; @@ -1291,7 +1270,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i #endif int ret; OVModel *ov_model = model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; TaskItem task; OVRequestItem *request; DNNExecBaseParams exec_params = { @@ -1308,7 +1287,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i } #if HAVE_OPENVINO2 - if (ctx->options.input_resizable) { + if (ctx->ov_option.input_resizable) { status = ov_partial_shape_create(4, dims, &partial_shape); if (status != OK) { av_log(ctx, AV_LOG_ERROR, "Failed to create partial shape.\n"); @@ -1339,7 +1318,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i if (!ov_model->compiled_model) { #else - if (ctx->options.input_resizable) { + if (ctx->ov_option.input_resizable) { status = ie_network_get_input_shapes(ov_model->network, &input_shapes); input_shapes.shapes->shape.dims[2] = input_height; input_shapes.shapes->shape.dims[3] = input_width; @@ -1386,11 +1365,10 @@ err: return ret; } -static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; OVModel *ov_model = NULL; - OVContext *ctx = NULL; #if HAVE_OPENVINO2 ov_core_t* core = NULL; ov_model_t* ovmodel = NULL; @@ -1411,17 +1389,9 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f av_freep(&model); return NULL; } + ov_model->ctx = ctx; model->model = ov_model; ov_model->model = model; - ov_model->ctx.class = &dnn_openvino_class; - ctx = &ov_model->ctx; - - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - goto err; - } #if HAVE_OPENVINO2 status = ov_core_create(&core); @@ -1430,13 +1400,13 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f } ov_model->core = core; - status = ov_core_read_model(core, model_filename, NULL, &ovmodel); + status = ov_core_read_model(core, ctx->model_filename, NULL, &ovmodel); if (status != OK) { ov_version_t ver; status = ov_get_openvino_version(&ver); av_log(NULL, AV_LOG_ERROR, "Failed to read the network from model file %s,\n" "Please check if the model version matches the runtime OpenVINO Version:\n", - model_filename); + ctx->model_filename); if (status == OK) { av_log(NULL, AV_LOG_ERROR, "BuildNumber: %s\n", ver.buildNumber); } @@ -1452,13 +1422,13 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f if (status != OK) goto err; - status = ie_core_read_network(ov_model->core, model_filename, NULL, &ov_model->network); + status = ie_core_read_network(ov_model->core, ctx->model_filename, NULL, &ov_model->network); if (status != OK) { ie_version_t ver; ver = ie_c_api_version(); av_log(ctx, AV_LOG_ERROR, "Failed to read the network from model file %s,\n" "Please check if the model version matches the runtime OpenVINO %s\n", - model_filename, ver.api_version); + ctx->model_filename, ver.api_version); ie_version_free(&ver); goto err; } @@ -1496,7 +1466,6 @@ static DNNModel *dnn_load_model_ov(const char *model_filename, DNNFunctionType f model->get_input = &get_input_ov; model->get_output = &get_output_ov; - model->options = options; model->filter_ctx = filter_ctx; model->func_type = func_type; @@ -1510,7 +1479,7 @@ err: static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_params) { OVModel *ov_model = model->model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; OVRequestItem *request; TaskItem *task; int ret; @@ -1539,7 +1508,7 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOMEM); } - ret = ff_dnn_fill_task(task, exec_params, ov_model, ctx->options.async, 1); + ret = ff_dnn_fill_task(task, exec_params, ov_model, ctx->async, 1); if (ret != 0) { av_freep(&task); return ret; @@ -1557,8 +1526,8 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return ret; } - if (ctx->options.async) { - while (ff_queue_size(ov_model->lltask_queue) >= ctx->options.batch_size) { + if (ctx->async) { + while (ff_queue_size(ov_model->lltask_queue) >= ctx->ov_option.batch_size) { request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); @@ -1581,7 +1550,7 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOSYS); } - if (ctx->options.batch_size > 1) { + if (ctx->ov_option.batch_size > 1) { avpriv_report_missing_feature(ctx, "batch mode for sync execution"); return AVERROR(ENOSYS); } @@ -1604,7 +1573,7 @@ static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, static int dnn_flush_ov(const DNNModel *model) { OVModel *ov_model = model->model; - OVContext *ctx = &ov_model->ctx; + DnnContext *ctx = ov_model->ctx; OVRequestItem *request; #if HAVE_OPENVINO2 ov_status_e status; @@ -1652,6 +1621,7 @@ static int dnn_flush_ov(const DNNModel *model) } const DNNModule ff_dnn_backend_openvino = { + .clazz = DNN_DEFINE_CLASS(dnn_openvino), .load_model = dnn_load_model_ov, .execute_model = dnn_execute_model_ov, .get_result = dnn_get_result_ov, diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 2ed17c3c87..d24591b90b 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -36,19 +36,8 @@ #include "safe_queue.h" #include -typedef struct TFOptions{ - char *sess_config; - uint8_t async; - uint32_t nireq; -} TFOptions; - -typedef struct TFContext { - const AVClass *class; - TFOptions options; -} TFContext; - -typedef struct TFModel{ - TFContext ctx; +typedef struct TFModel { + DnnContext *ctx; DNNModel *model; TF_Graph *graph; TF_Session *session; @@ -76,15 +65,13 @@ typedef struct TFRequestItem { DNNAsyncExecModule exec_module; } TFRequestItem; -#define OFFSET(x) offsetof(TFContext, x) +#define OFFSET(x) offsetof(TFOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_tensorflow_options[] = { - { "sess_config", "config for SessionOptions", OFFSET(options.sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, - DNN_BACKEND_COMMON_OPTIONS + { "sess_config", "config for SessionOptions", OFFSET(sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_tensorflow); static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue); static void infer_completion_callback(void *args); @@ -160,7 +147,7 @@ static int tf_start_inference(void *args) TFModel *tf_model = task->model; if (!request) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); + av_log(tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); return AVERROR(EINVAL); } @@ -170,7 +157,7 @@ static int tf_start_inference(void *args) task->nb_output, NULL, 0, NULL, request->status); if (TF_GetCode(request->status) != TF_OK) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); + av_log(tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); return DNN_GENERIC_ERROR; } return 0; @@ -198,7 +185,7 @@ static inline void destroy_request_item(TFRequestItem **arg) { static int extract_lltask_from_task(TaskItem *task, Queue *lltask_queue) { TFModel *tf_model = task->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; LastLevelTaskItem *lltask = av_malloc(sizeof(*lltask)); if (!lltask) { av_log(ctx, AV_LOG_ERROR, "Unable to allocate space for LastLevelTaskItem\n"); @@ -278,7 +265,7 @@ static TF_Tensor *allocate_input_tensor(const DNNData *input) static int get_input_tf(void *model, DNNData *input, const char *input_name) { TFModel *tf_model = model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TF_Status *status; TF_DataType dt; int64_t dims[4]; @@ -328,7 +315,7 @@ static int get_output_tf(void *model, const char *input_name, int input_width, i { int ret; TFModel *tf_model = model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TaskItem task; TFRequestItem *request; DNNExecBaseParams exec_params = { @@ -399,7 +386,7 @@ static int hex_to_data(uint8_t *data, const char *p) static int load_tf_model(TFModel *tf_model, const char *model_filename) { - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TF_Buffer *graph_def; TF_ImportGraphDefOptions *graph_opts; TF_SessionOptions *sess_opts; @@ -408,7 +395,7 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) int sess_config_length = 0; // prepare the sess config data - if (tf_model->ctx.options.sess_config != NULL) { + if (ctx->tf_option.sess_config != NULL) { const char *config; /* tf_model->ctx.options.sess_config is hex to present the serialized proto @@ -416,11 +403,11 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) proto in a python script, tools/python/tf_sess_config.py is a script example to generate the configs of sess_config. */ - if (strncmp(tf_model->ctx.options.sess_config, "0x", 2) != 0) { + if (strncmp(ctx->tf_option.sess_config, "0x", 2) != 0) { av_log(ctx, AV_LOG_ERROR, "sess_config should start with '0x'\n"); return AVERROR(EINVAL); } - config = tf_model->ctx.options.sess_config + 2; + config = ctx->tf_option.sess_config + 2; sess_config_length = hex_to_data(NULL, config); sess_config = av_mallocz(sess_config_length + AV_INPUT_BUFFER_PADDING_SIZE); @@ -461,7 +448,7 @@ static int load_tf_model(TFModel *tf_model, const char *model_filename) if (TF_GetCode(tf_model->status) != TF_OK) { TF_DeleteSessionOptions(sess_opts); av_log(ctx, AV_LOG_ERROR, "Failed to set config for sess options with %s\n", - tf_model->ctx.options.sess_config); + ctx->tf_option.sess_config); return DNN_GENERIC_ERROR; } } @@ -529,15 +516,14 @@ static void dnn_free_model_tf(DNNModel **model) TF_DeleteStatus(tf_model->status); } av_freep(&tf_model); - av_freep(model); + av_freep(&model); } } -static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; TFModel *tf_model = NULL; - TFContext *ctx = NULL; model = av_mallocz(sizeof(DNNModel)); if (!model){ @@ -551,23 +537,15 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f } model->model = tf_model; tf_model->model = model; - ctx = &tf_model->ctx; - ctx->class = &dnn_tensorflow_class; - - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - goto err; - } + tf_model->ctx = ctx; - if (load_tf_model(tf_model, model_filename) != 0){ - av_log(ctx, AV_LOG_ERROR, "Failed to load TensorFlow model: \"%s\"\n", model_filename); + if (load_tf_model(tf_model, ctx->model_filename) != 0){ + av_log(ctx, AV_LOG_ERROR, "Failed to load TensorFlow model: \"%s\"\n", ctx->model_filename); goto err; } - if (ctx->options.nireq <= 0) { - ctx->options.nireq = av_cpu_count() / 2 + 1; + if (ctx->nireq <= 0) { + ctx->nireq = av_cpu_count() / 2 + 1; } #if !HAVE_PTHREAD_CANCEL @@ -582,7 +560,7 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f goto err; } - for (int i = 0; i < ctx->options.nireq; i++) { + for (int i = 0; i < ctx->nireq; i++) { TFRequestItem *item = av_mallocz(sizeof(*item)); if (!item) { goto err; @@ -617,7 +595,6 @@ static DNNModel *dnn_load_model_tf(const char *model_filename, DNNFunctionType f model->get_input = &get_input_tf; model->get_output = &get_output_tf; - model->options = options; model->filter_ctx = filter_ctx; model->func_type = func_type; @@ -632,7 +609,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { LastLevelTaskItem *lltask; TaskItem *task; TFInferRequest *infer_request = NULL; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; int ret = 0; lltask = ff_queue_pop_front(tf_model->lltask_queue); @@ -728,7 +705,7 @@ static void infer_completion_callback(void *args) { DNNData *outputs; TFInferRequest *infer_request = request->infer_request; TFModel *tf_model = task->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; outputs = av_calloc(task->nb_output, sizeof(*outputs)); if (!outputs) { @@ -787,7 +764,7 @@ err: static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue) { TFModel *tf_model; - TFContext *ctx; + DnnContext *ctx; LastLevelTaskItem *lltask; TaskItem *task; int ret = 0; @@ -800,7 +777,7 @@ static int execute_model_tf(TFRequestItem *request, Queue *lltask_queue) lltask = ff_queue_peek_front(lltask_queue); task = lltask->task; tf_model = task->model; - ctx = &tf_model->ctx; + ctx = tf_model->ctx; ret = fill_model_input_tf(tf_model, request); if (ret != 0) { @@ -833,7 +810,7 @@ err: static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TaskItem *task; TFRequestItem *request; int ret = 0; @@ -849,7 +826,7 @@ static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_p return AVERROR(ENOMEM); } - ret = ff_dnn_fill_task(task, exec_params, tf_model, ctx->options.async, 1); + ret = ff_dnn_fill_task(task, exec_params, tf_model, ctx->async, 1); if (ret != 0) { av_log(ctx, AV_LOG_ERROR, "Fill task with invalid parameter(s).\n"); av_freep(&task); @@ -887,7 +864,7 @@ static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, static int dnn_flush_tf(const DNNModel *model) { TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + DnnContext *ctx = tf_model->ctx; TFRequestItem *request; int ret; @@ -915,6 +892,7 @@ static int dnn_flush_tf(const DNNModel *model) } const DNNModule ff_dnn_backend_tf = { + .clazz = DNN_DEFINE_CLASS(dnn_tensorflow), .load_model = dnn_load_model_tf, .execute_model = dnn_execute_model_tf, .get_result = dnn_get_result_tf, diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index ae55893a50..abdef1f178 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -36,18 +36,8 @@ extern "C" { #include "safe_queue.h" } -typedef struct THOptions{ - char *device_name; - int optimize; -} THOptions; - -typedef struct THContext { - const AVClass *c_class; - THOptions options; -} THContext; - typedef struct THModel { - THContext ctx; + DnnContext *ctx; DNNModel *model; torch::jit::Module *jit_model; SafeQueue *request_queue; @@ -67,20 +57,17 @@ typedef struct THRequestItem { } THRequestItem; -#define OFFSET(x) offsetof(THContext, x) +#define OFFSET(x) offsetof(THOptions, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_th_options[] = { - { "device", "device to run model", OFFSET(options.device_name), AV_OPT_TYPE_STRING, { .str = "cpu" }, 0, 0, FLAGS }, - { "optimize", "turn on graph executor optimization", OFFSET(options.optimize), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS}, + { "optimize", "turn on graph executor optimization", OFFSET(optimize), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, FLAGS}, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_th); - static int extract_lltask_from_task(TaskItem *task, Queue *lltask_queue) { THModel *th_model = (THModel *)task->model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; LastLevelTaskItem *lltask = (LastLevelTaskItem *)av_malloc(sizeof(*lltask)); if (!lltask) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for LastLevelTaskItem\n"); @@ -153,7 +140,6 @@ static void dnn_free_model_th(DNNModel **model) } ff_queue_destroy(th_model->task_queue); delete th_model->jit_model; - av_opt_free(&th_model->ctx); av_freep(&th_model); av_freep(model); } @@ -181,7 +167,7 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) TaskItem *task = NULL; THInferRequest *infer_request = NULL; DNNData input = { 0 }; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; int ret, width_idx, height_idx, channel_idx; lltask = (LastLevelTaskItem *)ff_queue_pop_front(th_model->lltask_queue); @@ -241,7 +227,7 @@ static int th_start_inference(void *args) LastLevelTaskItem *lltask = NULL; TaskItem *task = NULL; THModel *th_model = NULL; - THContext *ctx = NULL; + DnnContext *ctx = NULL; std::vector inputs; torch::NoGradGuard no_grad; @@ -253,9 +239,9 @@ static int th_start_inference(void *args) lltask = request->lltask; task = lltask->task; th_model = (THModel *)task->model; - ctx = &th_model->ctx; + ctx = th_model->ctx; - if (ctx->options.optimize) + if (ctx->torch_option.optimize) torch::jit::setGraphExecutorOptimize(true); else torch::jit::setGraphExecutorOptimize(false); @@ -292,7 +278,7 @@ static void infer_completion_callback(void *args) { outputs.dims[2] = sizes.at(2); // H outputs.dims[3] = sizes.at(3); // W } else { - avpriv_report_missing_feature(&th_model->ctx, "Support of this kind of model"); + avpriv_report_missing_feature(th_model->ctx, "Support of this kind of model"); goto err; } @@ -304,7 +290,7 @@ static void infer_completion_callback(void *args) { if (th_model->model->frame_post_proc != NULL) { th_model->model->frame_post_proc(task->out_frame, &outputs, th_model->model->filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, &outputs, &th_model->ctx); + ff_proc_from_dnn_to_frame(task->out_frame, &outputs, th_model->ctx); } } else { task->out_frame->width = outputs.dims[dnn_get_width_idx_by_layout(outputs.layout)]; @@ -312,7 +298,7 @@ static void infer_completion_callback(void *args) { } break; default: - avpriv_report_missing_feature(&th_model->ctx, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model->func_type); goto err; } task->inference_done++; @@ -322,7 +308,7 @@ err: if (ff_safe_queue_push_back(th_model->request_queue, request) < 0) { destroy_request_item(&request); - av_log(&th_model->ctx, AV_LOG_ERROR, "Unable to push back request_queue when failed to start inference.\n"); + av_log(th_model->ctx, AV_LOG_ERROR, "Unable to push back request_queue when failed to start inference.\n"); } } @@ -352,7 +338,7 @@ static int execute_model_th(THRequestItem *request, Queue *lltask_queue) goto err; } if (task->async) { - avpriv_report_missing_feature(&th_model->ctx, "LibTorch async"); + avpriv_report_missing_feature(th_model->ctx, "LibTorch async"); } else { ret = th_start_inference((void *)(request)); if (ret != 0) { @@ -375,7 +361,7 @@ static int get_output_th(void *model, const char *input_name, int input_width, i { int ret = 0; THModel *th_model = (THModel*) model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; TaskItem task = { 0 }; THRequestItem *request = NULL; DNNExecBaseParams exec_params = { @@ -424,12 +410,12 @@ static THInferRequest *th_create_inference_request(void) return request; } -static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx) +static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNModel *model = NULL; THModel *th_model = NULL; THRequestItem *item = NULL; - THContext *ctx; + const char *device_name = ctx->device ? ctx->device : "cpu"; model = (DNNModel *)av_mallocz(sizeof(DNNModel)); if (!model) { @@ -443,24 +429,17 @@ static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType f } th_model->model = model; model->model = th_model; - th_model->ctx.c_class = &dnn_th_class; - ctx = &th_model->ctx; - //parse options - av_opt_set_defaults(ctx); - if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { - av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - return NULL; - } + th_model->ctx = ctx; - c10::Device device = c10::Device(ctx->options.device_name); + c10::Device device = c10::Device(device_name); if (!device.is_cpu()) { - av_log(ctx, AV_LOG_ERROR, "Not supported device:\"%s\"\n", ctx->options.device_name); + av_log(ctx, AV_LOG_ERROR, "Not supported device:\"%s\"\n", device_name); goto fail; } try { th_model->jit_model = new torch::jit::Module; - (*th_model->jit_model) = torch::jit::load(model_filename); + (*th_model->jit_model) = torch::jit::load(ctx->model_filename); } catch (const c10::Error& e) { av_log(ctx, AV_LOG_ERROR, "Failed to load torch model\n"); goto fail; @@ -502,7 +481,6 @@ static DNNModel *dnn_load_model_th(const char *model_filename, DNNFunctionType f model->get_input = &get_input_th; model->get_output = &get_output_th; - model->options = NULL; model->filter_ctx = filter_ctx; model->func_type = func_type; return model; @@ -519,7 +497,7 @@ fail: static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_params) { THModel *th_model = (THModel *)model->model; - THContext *ctx = &th_model->ctx; + DnnContext *ctx = th_model->ctx; TaskItem *task; THRequestItem *request; int ret = 0; @@ -582,7 +560,7 @@ static int dnn_flush_th(const DNNModel *model) request = (THRequestItem *)ff_safe_queue_pop_front(th_model->request_queue); if (!request) { - av_log(&th_model->ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + av_log(th_model->ctx, AV_LOG_ERROR, "unable to get infer request.\n"); return AVERROR(EINVAL); } @@ -590,6 +568,7 @@ static int dnn_flush_th(const DNNModel *model) } extern const DNNModule ff_dnn_backend_torch = { + .clazz = DNN_DEFINE_CLASS(dnn_th), .load_model = dnn_load_model_th, .execute_model = dnn_execute_model_th, .get_result = dnn_get_result_th, diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index b9f71aea53..2d3bd229a4 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -24,12 +24,61 @@ */ #include "../dnn_interface.h" +#include "libavutil/avassert.h" #include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavfilter/internal.h" extern const DNNModule ff_dnn_backend_openvino; extern const DNNModule ff_dnn_backend_tf; extern const DNNModule ff_dnn_backend_torch; +#define OFFSET(x) offsetof(DnnContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM +static const AVOption dnn_base_options[] = { + {"model", "path to model file", + OFFSET(model_filename), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"input", "input name of the model", + OFFSET(model_inputname), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"output", "output name of the model", + OFFSET(model_outputnames_string), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {"backend_configs", "backend configs (deprecated)", + OFFSET(backend_options), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED}, + {"options", "backend configs (deprecated)", + OFFSET(backend_options), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED}, + {"nireq", "number of request", + OFFSET(nireq), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, FLAGS}, + {"async", "use DNN async inference", + OFFSET(async), AV_OPT_TYPE_BOOL, {.i64 = 1}, 0, 1, FLAGS}, + {"device", "device to run model", + OFFSET(device), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, FLAGS}, + {NULL} +}; + +AVFILTER_DEFINE_CLASS(dnn_base); + +typedef struct DnnBackendInfo { + const size_t offset; + union { + const AVClass *class; + const DNNModule *module; + }; +} DnnBackendInfo; + +static const DnnBackendInfo dnn_backend_info_list[] = { + {0, .class = &dnn_base_class}, + // Must keep the same order as in DNNOptions, so offset value in incremental order +#if CONFIG_LIBTENSORFLOW + {offsetof(DnnContext, tf_option), .module = &ff_dnn_backend_tf}, +#endif +#if CONFIG_LIBOPENVINO + {offsetof(DnnContext, ov_option), .module = &ff_dnn_backend_openvino}, +#endif +#if CONFIG_LIBTORCH + {offsetof(DnnContext, torch_option), .module = &ff_dnn_backend_torch}, +#endif +}; + const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) { switch(backend_type){ @@ -52,3 +101,43 @@ const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) return NULL; } } + +void ff_dnn_init_child_class(DnnContext *ctx) +{ + for (int i = 0; i < FF_ARRAY_ELEMS(dnn_backend_info_list); i++) { + const AVClass **ptr = (const AVClass **) ((char *) ctx + dnn_backend_info_list[i].offset); + *ptr = dnn_backend_info_list[i].class; + } +} + +void *ff_dnn_child_next(DnnContext *obj, void *prev) { + size_t pre_offset; + + if (!prev) { + av_assert0(obj->clazz); + return obj; + } + + pre_offset = (char *)prev - (char *)obj; + for (int i = 0; i < FF_ARRAY_ELEMS(dnn_backend_info_list) - 1; i++) { + if (dnn_backend_info_list[i].offset == pre_offset) { + const AVClass **ptr = (const AVClass **) ((char *) obj + dnn_backend_info_list[i + 1].offset); + av_assert0(*ptr); + return ptr; + } + } + + return NULL; +} + +const AVClass *ff_dnn_child_class_iterate(void **iter) +{ + uintptr_t i = (uintptr_t) *iter; + + if (i < FF_ARRAY_ELEMS(dnn_backend_info_list)) { + *iter = (void *)(i + 1); + return dnn_backend_info_list[i].class; + } + + return NULL; +} \ No newline at end of file diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 5e76b9ba45..860ca7591f 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -19,6 +19,7 @@ #include "dnn_filter_common.h" #include "libavutil/avstring.h" #include "libavutil/mem.h" +#include "libavutil/opt.h" #define MAX_SUPPORTED_OUTPUTS_NB 4 @@ -52,6 +53,23 @@ static char **separate_output_names(const char *expr, const char *val_sep, int * return parsed_vals; } +typedef struct DnnFilterBase { + const AVClass *class; + DnnContext dnnctx; +} DnnFilterBase; + +int ff_dnn_filter_init_child_class(AVFilterContext *filter) { + DnnFilterBase *base = filter->priv; + ff_dnn_init_child_class(&base->dnnctx); + return 0; +} + +void *ff_dnn_filter_child_next(void *obj, void *prev) +{ + DnnFilterBase *base = obj; + return ff_dnn_child_next(&base->dnnctx, prev); +} + int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { DNNBackendType backend = ctx->backend_type; @@ -91,7 +109,25 @@ int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *fil return AVERROR(EINVAL); } - ctx->model = (ctx->dnn_module->load_model)(ctx->model_filename, func_type, ctx->backend_options, filter_ctx); + if (ctx->backend_options) { + void *child = NULL; + + av_log(filter_ctx, AV_LOG_WARNING, + "backend_configs is deprecated, please set backend options directly\n"); + while (child = ff_dnn_child_next(ctx, child)) { + if (*(const AVClass **)child == &ctx->dnn_module->clazz) { + int ret = av_opt_set_from_string(child, ctx->backend_options, + NULL, "=", "&"); + if (ret < 0) { + av_log(filter_ctx, AV_LOG_ERROR, "failed to parse options \"%s\"\n", + ctx->backend_options); + return ret; + } + } + } + } + + ctx->model = (ctx->dnn_module->load_model)(ctx, func_type, filter_ctx); if (!ctx->model) { av_log(filter_ctx, AV_LOG_ERROR, "could not load DNN model\n"); return AVERROR(EINVAL); diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h index 30871ee381..b52b55a90d 100644 --- a/libavfilter/dnn_filter_common.h +++ b/libavfilter/dnn_filter_common.h @@ -26,28 +26,23 @@ #include "dnn_interface.h" -typedef struct DnnContext { - char *model_filename; - DNNBackendType backend_type; - char *model_inputname; - char *model_outputnames_string; - char *backend_options; - int async; - - char **model_outputnames; - uint32_t nb_outputs; - const DNNModule *dnn_module; - DNNModel *model; -} DnnContext; - -#define DNN_COMMON_OPTIONS \ - { "model", "path to model file", OFFSET(model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "input", "input name of the model", OFFSET(model_inputname), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "output", "output name of the model", OFFSET(model_outputnames_string), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "backend_configs", "backend configs", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "options", "backend configs (deprecated, use backend_configs)", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS | AV_OPT_FLAG_DEPRECATED},\ - { "async", "use DNN async inference (ignored, use backend_configs='async=1')", OFFSET(async), AV_OPT_TYPE_BOOL, { .i64 = 1}, 0, 1, FLAGS}, - +#define AVFILTER_DNN_DEFINE_CLASS_EXT(name, desc, options) \ + static const AVClass name##_class = { \ + .class_name = desc, \ + .item_name = av_default_item_name, \ + .option = options, \ + .version = LIBAVUTIL_VERSION_INT, \ + .category = AV_CLASS_CATEGORY_FILTER, \ + .child_next = ff_dnn_filter_child_next, \ + .child_class_iterate = ff_dnn_child_class_iterate, \ + } + +#define AVFILTER_DNN_DEFINE_CLASS(fname) \ + AVFILTER_DNN_DEFINE_CLASS_EXT(fname, #fname, fname##_options) + +void *ff_dnn_filter_child_next(void *obj, void *prev); + +int ff_dnn_filter_init_child_class(AVFilterContext *filter); int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); int ff_dnn_set_frame_proc(DnnContext *ctx, FramePrePostProc pre_proc, FramePrePostProc post_proc); diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 63f492e690..4e544486cc 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -93,8 +93,6 @@ typedef int (*ClassifyPostProc)(AVFrame *frame, DNNData *output, uint32_t bbox_i typedef struct DNNModel{ // Stores model that can be different for different backends. void *model; - // Stores options when the model is executed by the backend - const char *options; // Stores FilterContext used for the interaction between AVFrame and DNNData AVFilterContext *filter_ctx; // Stores function type of the model @@ -117,10 +115,65 @@ typedef struct DNNModel{ ClassifyPostProc classify_post_proc; } DNNModel; +typedef struct TFOptions{ + const AVClass *clazz; + + char *sess_config; +} TFOptions; + +typedef struct OVOptions { + const AVClass *clazz; + + int batch_size; + int input_resizable; + DNNLayout layout; + float scale; + float mean; +} OVOptions; + +typedef struct THOptions { + const AVClass *clazz; + int optimize; +} THOptions; + +typedef struct DNNModule DNNModule; + +typedef struct DnnContext { + const AVClass *clazz; + + DNNModel *model; + + char *model_filename; + DNNBackendType backend_type; + char *model_inputname; + char *model_outputnames_string; + char *backend_options; + int async; + + char **model_outputnames; + uint32_t nb_outputs; + const DNNModule *dnn_module; + + int nireq; + char *device; + +#if CONFIG_LIBTENSORFLOW + TFOptions tf_option; +#endif + +#if CONFIG_LIBOPENVINO + OVOptions ov_option; +#endif +#if CONFIG_LIBTORCH + THOptions torch_option; +#endif +} DnnContext; + // Stores pointers to functions for loading, executing, freeing DNN models for one of the backends. -typedef struct DNNModule{ +struct DNNModule { + const AVClass clazz; // Loads model and parameters from given file. Returns NULL if it is not possible. - DNNModel *(*load_model)(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); + DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); // Executes model with specified input and output. Returns the error code otherwise. int (*execute_model)(const DNNModel *model, DNNExecBaseParams *exec_params); // Retrieve inference result. @@ -129,11 +182,15 @@ typedef struct DNNModule{ int (*flush)(const DNNModel *model); // Frees memory allocated for model. void (*free_model)(DNNModel **model); -} DNNModule; +}; // Initializes DNNModule depending on chosen backend. const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx); +void ff_dnn_init_child_class(DnnContext *ctx); +void *ff_dnn_child_next(DnnContext *obj, void *prev); +const AVClass *ff_dnn_child_class_iterate(void **iter); + static inline int dnn_get_width_idx_by_layout(DNNLayout layout) { return layout == DL_NHWC ? 2 : 3; diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index c8848dd7ba..7f665b73ab 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -46,13 +46,10 @@ static const AVOption derain_options[] = { #if (CONFIG_LIBTENSORFLOW == 1) { "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, .unit = "backend" }, #endif - { "model", "path to model file", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, - { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(derain); +AVFILTER_DNN_DEFINE_CLASS(derain); static int filter_frame(AVFilterLink *inlink, AVFrame *in) { @@ -113,6 +110,7 @@ const AVFilter ff_vf_derain = { .name = "derain", .description = NULL_IF_CONFIG_SMALL("Apply derain filter to the input."), .priv_size = sizeof(DRContext), + .preinit = ff_dnn_filter_init_child_class, .init = init, .uninit = uninit, FILTER_INPUTS(derain_inputs), diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index 1f8f227e3a..965779a8ab 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -50,14 +50,13 @@ static const AVOption dnn_classify_options[] = { #if (CONFIG_LIBOPENVINO == 1) { "openvino", "openvino backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" }, #endif - DNN_COMMON_OPTIONS { "confidence", "threshold of confidence", OFFSET2(confidence), AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS}, { "labels", "path to labels file", OFFSET2(labels_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { "target", "which one to be classified", OFFSET2(target), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_classify); +AVFILTER_DNN_DEFINE_CLASS(dnn_classify); static int dnn_classify_post_proc(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx) { @@ -299,6 +298,7 @@ const AVFilter ff_vf_dnn_classify = { .name = "dnn_classify", .description = NULL_IF_CONFIG_SMALL("Apply DNN classify filter to the input."), .priv_size = sizeof(DnnClassifyContext), + .preinit = ff_dnn_filter_init_child_class, .init = dnn_classify_init, .uninit = dnn_classify_uninit, FILTER_INPUTS(ff_video_default_filterpad), diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index bacea3ef29..926966368a 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -70,7 +70,6 @@ static const AVOption dnn_detect_options[] = { #if (CONFIG_LIBOPENVINO == 1) { "openvino", "openvino backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" }, #endif - DNN_COMMON_OPTIONS { "confidence", "threshold of confidence", OFFSET2(confidence), AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS}, { "labels", "path to labels file", OFFSET2(labels_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { "model_type", "DNN detection model type", OFFSET2(model_type), AV_OPT_TYPE_INT, { .i64 = DDMT_SSD }, INT_MIN, INT_MAX, FLAGS, .unit = "model_type" }, @@ -85,7 +84,7 @@ static const AVOption dnn_detect_options[] = { { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_detect); +AVFILTER_DNN_DEFINE_CLASS(dnn_detect); static inline float sigmoid(float x) { return 1.f / (1.f + exp(-x)); @@ -851,6 +850,7 @@ const AVFilter ff_vf_dnn_detect = { .name = "dnn_detect", .description = NULL_IF_CONFIG_SMALL("Apply DNN detect filter to the input."), .priv_size = sizeof(DnnDetectContext), + .preinit = ff_dnn_filter_init_child_class, .init = dnn_detect_init, .uninit = dnn_detect_uninit, FILTER_INPUTS(dnn_detect_inputs), diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index fdac31665e..9a1dd2a356 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -54,11 +54,10 @@ static const AVOption dnn_processing_options[] = { #if (CONFIG_LIBTORCH == 1) { "torch", "torch backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_TH }, 0, 0, FLAGS, "backend" }, #endif - DNN_COMMON_OPTIONS { NULL } }; -AVFILTER_DEFINE_CLASS(dnn_processing); +AVFILTER_DNN_DEFINE_CLASS(dnn_processing); static av_cold int init(AVFilterContext *context) { @@ -373,6 +372,7 @@ const AVFilter ff_vf_dnn_processing = { .name = "dnn_processing", .description = NULL_IF_CONFIG_SMALL("Apply DNN processing filter to the input."), .priv_size = sizeof(DnnProcessingContext), + .preinit = ff_dnn_filter_init_child_class, .init = init, .uninit = uninit, FILTER_INPUTS(dnn_processing_inputs), diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 60683b5209..f14c0c0cd3 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -50,13 +50,10 @@ static const AVOption sr_options[] = { { "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, .unit = "backend" }, #endif { "scale_factor", "scale factor for SRCNN model", OFFSET(scale_factor), AV_OPT_TYPE_INT, { .i64 = 2 }, 2, 4, FLAGS }, - { "model", "path to model file specifying network architecture and its parameters", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, - { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } }; -AVFILTER_DEFINE_CLASS(sr); +AVFILTER_DNN_DEFINE_CLASS(sr); static av_cold int init(AVFilterContext *context) { @@ -192,6 +189,7 @@ const AVFilter ff_vf_sr = { .name = "sr", .description = NULL_IF_CONFIG_SMALL("Apply DNN-based image super resolution to the input."), .priv_size = sizeof(SRContext), + .preinit = ff_dnn_filter_init_child_class, .init = init, .uninit = uninit, FILTER_INPUTS(sr_inputs), From patchwork Tue Apr 30 07:12:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48424 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp322739pza; Tue, 30 Apr 2024 00:13:07 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWYcvrF1PdAzabbt8L02CfMcPLC7v5KQjN1fxyboXRMt65y1jkArIyhMsfukQMvm5gEINn6ROQfI9K1C5Abpa7MUiyKC2rRmQu4Kg== X-Google-Smtp-Source: AGHT+IHNaZYa6cSRlojt5pZMr4WIhBqwFQy+Hrv1yQX4VE6rLUpY0TkrAn8PiQ5f4LwsjNDqlzzc X-Received: by 2002:a05:6402:3222:b0:572:5dcd:a68 with SMTP id g34-20020a056402322200b005725dcd0a68mr9009635eda.4.1714461187299; Tue, 30 Apr 2024 00:13:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461187; cv=none; d=google.com; s=arc-20160816; b=PiAQuNzgQd61+ZHIYjcoxPDcIdF/D01D86wajeUD2xtHfhg5k4pGZyf+hhsyYX17FV rOZwrHTZOyQvvWzmtiH56D2PiTH1Ir/lhAcFFEQ8ulV3GHqX5ZLwS3ElYGx1hE64bVJj bmB2Iz2IoYnl87zOcJt9Smk10YYLQ0sOYGry9/qA3i0NkML4zG5NBLCRkkNZQ6VeK35S xXDxnsmRNpHeovg8GRg08YotWQFRQ9b4wjLdXJRqqEuzN6aiWQ2uV9n9SPXup1Oh2nJL gLP/ZpDze7vte/OBwU5HmmIgSIyIW6RAjlsD7HMeIrW7aS0OIu9rtYaAaTWkGTkstJu9 8ZxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=keuiO01UiEPaQ56y9U9tydp78EvLfvfowOBsqmbTUMs=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=gxRjxLCkay4ZWyPX7izPR5WHDNIGO0jIYM4rudc+znHn+yOxXMN08ONQy6UFDKzIET uLrsXa+d+s+MA34gfymPWHWq5dWYKoAecRbbt4pcvO6Fkl/Vd6jmZ38No81nvVxzxyXJ iYszlFwO+ocDTyVp2TTnz95d6kSa0UEqEGIsjflawS9oZNHBkgSmxp3x7lA6OByueH02 2ES61qhTrORAnAE/pWQIX+GTOk8yokVQOvRTcLRCLxpX/1IijTupz8O5bid+j5ZpSW3l PHtgwahxPbEPKK4cs6yPVneHzrIo2AeXKDTRIkGpPK8q3j7xIC94w/j3ov2TK8oc//T2 QVLA==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="RrYKCxu/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m13-20020a056402430d00b005729a4e1e28si305636edc.253.2024.04.30.00.13.06; Tue, 30 Apr 2024 00:13:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="RrYKCxu/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AF2A868D16A; Tue, 30 Apr 2024 10:12:36 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-49.mail.qq.com (out162-62-57-49.mail.qq.com [162.62.57.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4AE2868D59E for ; Tue, 30 Apr 2024 10:12:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461134; bh=e51EtxHOu+KQP06rtPnENZW/tXAvLTbCtDASrgZximU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=RrYKCxu/YIVoIwqzyFpoAaxrlAIfJApswuIY4/cj1eLSeo18/a5pqdAUVaV/2U/1k lzHHnM/clC5No+FseHiIA6eEIWai4ICqgLwZEPwFg1F8d2XyQAt6KP8nl1siDTA7tY IoCOEswnI9iHZvmKGCCuffiNX5xveap5zToRQz9k= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461133tf32kd32e Message-ID: X-QQ-XMAILINFO: NMGzQWUSIfvT4T/o2vUUi8NAXlY8TqRJKr80XROVltWJHqLlfIOFose8eAFukp Li8j/EFKSWdaI6GIC96YHR+oLHsctrNzXzfXoJpEbfQnsCGKSrj4P8xCgPz2e+ue8FsHNwTw4vkM z7f4gonSyp8A+wF/GZWMjHvj95+gRt5wSULM5uPGqrJjITOzFhW5hq53+gl3lokfyO2EKXlgn5tF eoxSZXKp4LX/8dXz/BkDCIsZxFWcFlODK2QLFbjAjfo/zHXIMSve91RbcWZiemV6o5lmVYkljJ2v Yr1O+yGTZw8WBLNJGwhk5TmBfO2/pm2HtWGStairaTpXuGUlJLY2jUzvZ5kown+0yFDUTmw48c9m cCTV8Z6NJTo8+xNU+SdWv+g6iTG8+D9cNTO1Yb+5WQlS/zHhsrQDhVB3NKFMAPBFLezy8QTDc5sj ACMrYYSekRGC1RkYIRbkx+W1JO8t2hbH5avAquZGGULk5RGBA+4FiSE4iPWb1xMUdC2ulvG75Y40 bGE91mHJLg6ZwZ6vciHBOregG8s+BIW/OWzrvT1GYAcanieDem02v9rVbgzDUpPRCx6q0+7mb6pH WJRCq19aWDUbsYM4xw/l7P8+hmE9Tv+gyKshblfAs6pg2Mh6y0/ugP/VvvU7IPQosw8VRa92s5tB Zcr/LbtNrUWbB5dKA14awxTLSpZVxAj/NGquPzd3AnuCKf0A/zvgl+RFn3jm6ns45RKR8VBjKkKo D0ai5rFy9ROmER+1xMZJ/jt5M0h3srdK8sBZLE2iWuodduXCiQ+MeiaYOvdihUbmDvrvadHrK5B8 O3UKIV8CtUYqOV1K6Iwg6JFb3nZMAQvphSGQu/OyCJegETaU5SODX0+rdQG8CMAbugSTSfsjZ/+Y RggIxlNp8EmmAtWgbtAIj6AVt2B2kUlbUsOFYXKdnd0xQyq5dd5zBOaNTwJX5fVV/VY51Dd0Rhir uGR6sGNwIRCqIWNektVR/lyKYBzU6jQkHvemUOG4IhT7q9QrtMRRwRd0qa3oyK X-QQ-XMRINFO: OWPUhxQsoeAVDbp3OJHYyFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:00 +0800 X-OQ-MSGID: <20240430071208.126817-3-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 02/10] avfilter/dnn_backend_openvino: Fix free context at random place X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: V96u9EGCphv+ From: Zhao Zhili It will be freed again by ff_dnn_uninit. --- libavfilter/dnn/dnn_backend_openvino.c | 1 - 1 file changed, 1 deletion(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index c4b0682f11..769ba0a54b 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -959,7 +959,6 @@ err: if (input_model_info) ov_preprocess_input_model_info_free(input_model_info); #endif - dnn_free_model_ov(&ov_model->model); return ret; } From patchwork Tue Apr 30 07:12:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48422 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp322361pza; Tue, 30 Apr 2024 00:12:46 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVuKtX2GXJsS3Fuc9PF/KGtHqhCqxHsIOCNUlaIM138Hy5JIrnxdFkBAv78gvoNpMvwqzFlC15OCSA79+LxYoIXh3mIOLQLnrHCZQ== X-Google-Smtp-Source: AGHT+IFUs1zZ2MT+Y6NgQbRVWJkmLX2MiPHU4Iw40UeaexnK90YscUAFgaJOgndh4sMcPeAESCPV X-Received: by 2002:a17:906:3c03:b0:a58:c5a5:209b with SMTP id h3-20020a1709063c0300b00a58c5a5209bmr1765973ejg.24.1714461166474; Tue, 30 Apr 2024 00:12:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461166; cv=none; d=google.com; s=arc-20160816; b=vib8Q8qP2OAZikELltM2nPOjFS+2ZvmFjASBdEQWjvGQlA34ssnj9EGMcAJ2mMGhpB RfLNBDEBPYb5dU0hDxmLcOM/eilUcpykzzJ/1h7f6Y7FBRQxrGveH7XTP2fxqkod5YXS pqcj2ndenXWx7EOKKAG/vUC+BxRq0A8LSTUaHcvgzEV4D+kHyDuyymWwqIUd/MdyTxJc xE84j1GzeFPLpSXRylxpMDnnx6pqjwI0BNfl+azSzTu4gemZ0Qz5Z7qr0L8b75SqR0m6 zViG+u2k+bciVsnLReiuhhCX366Z1TTMlOgXS9az7Hz/ANu0FQWKm5UlcNRUnVG+Hy3H j91g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=q28SO0LScS/xtFl5jFyMixzXAjjCm2BSdG0vh4vwGXg=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=OpehHAOtdp0puZmE2owb3NOhcErPZ6OuNvhP/YTvrims08yRXqatSQZ8lnv64ze1fE xlHnxJva+BcOjWTVw48+SCwdvbbVggzklxIIJeRExuXn2C2zmpIGH7updj2DoBrIxL// 3Imi+dZSthSixVsM+MDUfRH/16OnZkxNPH1LBY5FdcP5KaHGRlwQx5xAWtlR/nNkafoE C4iVP6SyAF3e/k1I824NzkNjkcNQzO3BTUYtz9gun67hkZAn60Ae8LjFbr+63hXftIeO fUI1gInSEGZQeAPmDW+0gw3P/foZ1LGYcM9sEDOIgbqUzZKVVmioL8gUOBQxk7Y+ZhZU 3oeg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=T9GDfzTU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ck25-20020a170906c45900b00a587ed71f6bsi8895233ejb.966.2024.04.30.00.12.45; Tue, 30 Apr 2024 00:12:46 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=T9GDfzTU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 596B268D541; Tue, 30 Apr 2024 10:12:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-84.mail.qq.com (unknown [203.205.251.84]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 694C868D220 for ; Tue, 30 Apr 2024 10:12:23 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461134; bh=HAukkNZJvEOwRRpmfNHUU5zmhF6whRNbp+SKycaRzbs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=T9GDfzTUpFJD48/bKVt+KMQZpLS31mCfhXpCUuTglBBU7mSzbO6awfxsEhHPB4jpA zSiP57yHYZNTgTOGOYDxKvJHITwOnJMS0zUWaEGH4Yyi6QuZhqtFZSOOedKN0XOQwk 3nU/GG9iiPg8Aq1yzOeG195rvOEJdILcIV/jHq6k= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461134tq39cub2j Message-ID: X-QQ-XMAILINFO: MmpliBmRb3iCcN1WUoFrt4KNLOHL29dnnWIy0Rq46mIm+OaT0Qpsd6rv+MCWOz Nd7S85TMlvkt3htRheneJuKG4wvG68FVkxWOO18B2ySXHr+wu9CFyz5if/cSnOa1r2BNQQMGxurq Bzh7yYbMVUDQWsEw3CZ+LeusElAdqU92mL3orykTJGMBq7EUbyeAKEm5pSeVodsRiM8tuikaCx7q 2MLi7YncshEclYCclEuHKEl0qBwwKc+o8CumIMuVm7kxVlwfV9yyN0Dk2NdnjR9iP7h5Ok7X/2RH MHGcXf1I3czrnvNhbRONcScK2qb2ZVkmIuq+Tpt/hyl6d9KShaiz+ixR8TxJXNcE2hPwhFm3hfa+ ZnkC+xTW+xhiN0AnnS1NM6H10MoTRoZ/HcnKDeut5lwKv/0W2Qg1Bv0mYyYxtvbt2vBd7GLcLrxv NdkV7M1xXq+IJDDcpN45bgYKnDG5OuGdXPVSfvk9pjVh6/SaTLwzbHZ6XZSpedY106TjwuTfxwA6 G/KINLkbWwiUqzDTt9OqxFDYGgIPsSAk8C9r2vnSJcdQD8k76p2SLdpmeQq+EWsFxUMICJr/LjFK GIOZuyDroio4vtGSFzZX4xFBjMG/huZoD4QaN5DN+hQ7d8iidWLKIfyPSEOCXaysaTpTJ3pFwi1Y De2E3zpK5cxkY7f/wnQ8sId1a8Y6/5kh47xxH17y6Nnl7WBstaz2YFcHnGS1IztPkIAg6XT3M113 qs1SqPsgyRjGtpu5pBRkeJGn9FIStWWUBA6iGlb55svOPTk46q1DIl1rN4YVQCh4GMNha1rw2RVg 98aBIqWc/Jam9GdWZ66bCI3eow8DOSf/PO3DhTR2tyItKPssTd7089E4s4WrYY6UAO9V4yCiKmjg 0y5Vyfko0BVyF6MSHgJ+m5NFlqIjE6CBE0An4iOYwJqsUTtvzgX5Hi9JvYUSI4L9UzKRQjw0zSho n++05rFTmMdBq0hOVSqYyhVPNyTR7+RAnZ08XdviNgqGmI4GWc70ePWCRp1+gf X-QQ-XMRINFO: NyFYKkN4Ny6FSmKK/uo/jdU= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:01 +0800 X-OQ-MSGID: <20240430071208.126817-4-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 03/10] avfilter/dnn_backend_openvino: simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jnEHFJIPxWos From: Zhao Zhili --- libavfilter/dnn/dnn_backend_openvino.c | 47 +++++++++++--------------- 1 file changed, 20 insertions(+), 27 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 769ba0a54b..1acc54b791 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -41,8 +41,8 @@ #include "dnn_backend_common.h" typedef struct OVModel{ + DNNModel model; DnnContext *ctx; - DNNModel *model; #if HAVE_OPENVINO2 ov_core_t *core; ov_model_t *ov_model; @@ -300,11 +300,11 @@ static int fill_model_input_ov(OVModel *ov_model, OVRequestItem *request) return ov2_map_error(status, NULL); } #endif - switch (ov_model->model->func_type) { + switch (ov_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (ov_model->model->frame_pre_proc != NULL) { - ov_model->model->frame_pre_proc(task->in_frame, &input, ov_model->model->filter_ctx); + if (ov_model->model.frame_pre_proc != NULL) { + ov_model->model.frame_pre_proc(task->in_frame, &input, ov_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } @@ -442,11 +442,11 @@ static void infer_completion_callback(void *args) for (int i = 0; i < request->lltask_count; ++i) { task = request->lltasks[i]->task; - switch (ov_model->model->func_type) { + switch (ov_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (ov_model->model->frame_post_proc != NULL) { - ov_model->model->frame_post_proc(task->out_frame, outputs, ov_model->model->filter_ctx); + if (ov_model->model.frame_post_proc != NULL) { + ov_model->model.frame_post_proc(task->out_frame, outputs, ov_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } @@ -458,23 +458,23 @@ static void infer_completion_callback(void *args) } break; case DFT_ANALYTICS_DETECT: - if (!ov_model->model->detect_post_proc) { + if (!ov_model->model.detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); goto end; } - ov_model->model->detect_post_proc(task->in_frame, outputs, + ov_model->model.detect_post_proc(task->in_frame, outputs, ov_model->nb_outputs, - ov_model->model->filter_ctx); + ov_model->model.filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: - if (!ov_model->model->classify_post_proc) { + if (!ov_model->model.classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); goto end; } for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++) - ov_model->model->classify_post_proc(task->in_frame, outputs, + ov_model->model.classify_post_proc(task->in_frame, outputs, request->lltasks[i]->bbox_index, - ov_model->model->filter_ctx); + ov_model->model.filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -571,7 +571,7 @@ static void dnn_free_model_ov(DNNModel **model) av_free(ov_model->all_input_names); #endif av_freep(&ov_model); - av_freep(model); + *model = NULL; } @@ -598,7 +598,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * #endif // We scale pixel by default when do frame processing. if (fabsf(ctx->ov_option.scale) < 1e-6f) - ctx->ov_option.scale = ov_model->model->func_type == DFT_PROCESS_FRAME ? 255 : 1; + ctx->ov_option.scale = ov_model->model.func_type == DFT_PROCESS_FRAME ? 255 : 1; // batch size if (ctx->ov_option.batch_size <= 0) { ctx->ov_option.batch_size = 1; @@ -702,7 +702,7 @@ static int init_model_ov(OVModel *ov_model, const char *input_name, const char * ret = ov2_map_error(status, NULL); goto err; } - if (ov_model->model->func_type != DFT_PROCESS_FRAME) + if (ov_model->model.func_type != DFT_PROCESS_FRAME) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); else if (fabsf(ctx->ov_option.scale - 1) > 1e-6f || fabsf(ctx->ov_option.mean) > 1e-6f) status |= ov_preprocess_output_set_element_type(output_tensor_info, F32); @@ -1280,7 +1280,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i .out_frame = NULL, }; - if (ov_model->model->func_type != DFT_PROCESS_FRAME) { + if (ov_model->model.func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); return AVERROR(EINVAL); } @@ -1342,7 +1342,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i goto err; } - ret = extract_lltask_from_task(ov_model->model->func_type, &task, ov_model->lltask_queue, NULL); + ret = extract_lltask_from_task(ov_model->model.func_type, &task, ov_model->lltask_queue, NULL); if (ret != 0) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); goto err; @@ -1378,19 +1378,12 @@ static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, A IEStatusCode status; #endif - model = av_mallocz(sizeof(DNNModel)); - if (!model){ - return NULL; - } - ov_model = av_mallocz(sizeof(OVModel)); - if (!ov_model) { - av_freep(&model); + if (!ov_model) return NULL; - } ov_model->ctx = ctx; + model = &ov_model->model; model->model = ov_model; - ov_model->model = model; #if HAVE_OPENVINO2 status = ov_core_create(&core); From patchwork Tue Apr 30 07:12:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48423 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp322561pza; Tue, 30 Apr 2024 00:12:57 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXjoBe6Bigam9PpwDgHLI2k8qTq4/LhXWbY8/ntwxzpuVkgc6pP2QRXrPXfydIHz+tppZ3gmsIcrCT4A9jp1kGVgv4R2NP82UciKw== X-Google-Smtp-Source: AGHT+IHBxgpJNFvojcWvjFtqJi0FC1EVnzWJF+wq/96p5e0/qh3qtvBgzz4w/PQ5hQzQgRHlVzDJ X-Received: by 2002:a17:906:5a51:b0:a59:2065:13d7 with SMTP id my17-20020a1709065a5100b00a59206513d7mr1361414ejc.28.1714461177558; Tue, 30 Apr 2024 00:12:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461177; cv=none; d=google.com; s=arc-20160816; b=AwLadjVWJP5Adv9/WEu547l2HZYnVbtTLZACveoiFJRr63qDvlG8UO5dmiQctI1xvv STzfIkZ4Jd1RgNdc7ORQ+RkU/vCNmuesG086NEYX4LAc/MpNhOekFybCTAE7daJlf8uP mFTaRAUH1EcbiHEyqb18qCtqXUCj1gMPhCOZa/skDZLdmmRfC45BPnzfY8XpUVL4po8y cI/7NQeFIdEqYsPRkuNhgUUzvZxiqjdM6VJpHGdU01Rj0x27LDYHhABP6CHQ3M4B8/z6 8fwv9fiq8shtOtbGsLD702nEY9sc3xDnmVtzqNBr9CDFBV3ANYtRVdoUNcbPK/9/rYJz cpmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=4J0uSrESCcvYgXCjegzZhhawMSin0wKS9wlbV007N1c=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=oXxV5OSKiVnvhx9dKm4ePo+wFhEE9eJYdyDrR/GZS/ejb2jL/7EG+ZoDC8vglnPLGU 84CBryj2//w3wLFWApU58G6HclGZPzE7alqD1J++dLJG9WjniT+uYxDxb2gVYUCUvroz lU44Ov+Avw1R6Fi8yBzJMcIaBUkFKw2rWbIPsdz50qZjYNaqR8rpr25cOYJV7vCem4FL pwFRXsZsvX4GoG3HfqPOXUBMGkJVZTPv106Qz8aSMVnjhzCFV5fqeCN4b7l0H2QchUPW V9puQzPdR9Jd5Wyouh7ERCt956ggzz8S7WE7CX+e6z0uj0AY97IlkJoQXwbsqaKnMBKu g++Q==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=Rgv2rHou; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o21-20020a17090608d500b00a524d96b9cesi14371150eje.11.2024.04.30.00.12.56; Tue, 30 Apr 2024 00:12:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=Rgv2rHou; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E943068D5D1; Tue, 30 Apr 2024 10:12:34 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-59.mail.qq.com (out203-205-251-59.mail.qq.com [203.205.251.59]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C034868D582 for ; Tue, 30 Apr 2024 10:12:23 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461135; bh=1wklaRLE14l6xLaWgU45zthbEJNQvCoOjvvidDHrzgc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Rgv2rHouwAJ1Uj+q6LYRqdtZoQz9bLUcd3bq18e343/WYocdmuyrtc/2AdLnFqNTy 0sq81CRHtZB31QUGIgArGZrEGQSdKCcRnsPLswIKj8iFQab0txn7cCrNlHaiSsmyPN Cz/avCYwxRYD6mM+/loIm1WCj+dY24utTPxIESxE= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461134tgmsclvf2 Message-ID: X-QQ-XMAILINFO: M7uElAZZZMmFED9XRmM5uTKVs/Ztq/BJmh6YbZ7UeOks6+kD11yzX1ZixwVRc6 n9yTBEeXL2FjJSH89MCis5S/cwOu637DKXiH4xpHJ6li0ci1HhsbZC+lis1vVbk5sv8QyqH+AMYb fv3GJj63KiSHj7Lw9JZcXI3oLHJ91CIoimxoFucZN64mSTKBdSWiI8cu791vamWi9IVP+/iD/fC3 e1jD9A2HUlLHsfUrPvqBZkRnN9yiZhBw1PLoYg/sgmFw0t2WKvUBkuYxpoHX/HFUR7EGEYgdrmb5 TPpBt9AocArUc4bMbxq/Q35e08SCIIC76V15BQ1azSbEyBGZnLidAQP5x5t2yIiYp/QubwSD3xZ4 end7eC2Q3txFGKRWFQgdTrOEHBTVwtMuuhxbOA/ty21UXLp42GCfJzngGFDFQ5E/lEIFadI99IE6 ijR4QAU0JQ0raaKpButjUNg970bkV+sIgZAq840nWBLEX8t13TGDtTtvSJYS4TwaSl2Tdpkao3Jf KO/LlzMJx/q8TUenmy3Pf0cTeS0zbkZ3DAAD2+n4BVi9q8NrzmzeAl9qRP0QjeXAMUeZPDORbsMZ p1BgEp6XKx0O9NaDZk6CvPFsl7aaBJKtMnAjJO/5uthz3SJoWqCfIulfB8uxCd4chL7ElfdpcmYX usBVJ5BOeqccR3DGPeajyNZEPYrE0TJl01OGWDgsni+GacQnVlBnf0w6KA7MPiNX+pK2cTHKNPe8 QDQ6SthxQ3/S/FGgkJ68Fg7qeKg0sYbJOudihbJk1OE1JNsGpyLdxHpEp/lIGP20Pt5b/COuVajs hw74sEHxpbLJE4aTvzToXhUdSraESdgWjZdCAkCOMert40XtZMPknkcj6G705xDGyhlJnrKhWSK2 QGwV4ZAjyxxTH3a7zC4xvbL5aRw4hnbdMeIjuZVZ2Gu7WXz0spJT2lWI1lA7KNkiFZ2J5h4dsW2/ RooFydd209Tcllr/100moBeb6a17k//x9REl1APxhUS5m9lLlrXD0NTHA9xZDINEP+v8RSqcRw5k uH88G8ljdqR+QEGbOA X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:02 +0800 X-OQ-MSGID: <20240430071208.126817-5-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 04/10] avfilter/dnn_backend_tf: Remove one level of indentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: JwOYhWzfhhvs From: Zhao Zhili --- libavfilter/dnn/dnn_backend_tf.c | 63 ++++++++++++++++---------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index d24591b90b..60f9e57fb7 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -483,41 +483,42 @@ static void dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; - if (*model){ - tf_model = (*model)->model; - while (ff_safe_queue_size(tf_model->request_queue) != 0) { - TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); - destroy_request_item(&item); - } - ff_safe_queue_destroy(tf_model->request_queue); + if (!model || !*model) + return; - while (ff_queue_size(tf_model->lltask_queue) != 0) { - LastLevelTaskItem *item = ff_queue_pop_front(tf_model->lltask_queue); - av_freep(&item); - } - ff_queue_destroy(tf_model->lltask_queue); + tf_model = (*model)->model; + while (ff_safe_queue_size(tf_model->request_queue) != 0) { + TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + destroy_request_item(&item); + } + ff_safe_queue_destroy(tf_model->request_queue); - while (ff_queue_size(tf_model->task_queue) != 0) { - TaskItem *item = ff_queue_pop_front(tf_model->task_queue); - av_frame_free(&item->in_frame); - av_frame_free(&item->out_frame); - av_freep(&item); - } - ff_queue_destroy(tf_model->task_queue); + while (ff_queue_size(tf_model->lltask_queue) != 0) { + LastLevelTaskItem *item = ff_queue_pop_front(tf_model->lltask_queue); + av_freep(&item); + } + ff_queue_destroy(tf_model->lltask_queue); - if (tf_model->graph){ - TF_DeleteGraph(tf_model->graph); - } - if (tf_model->session){ - TF_CloseSession(tf_model->session, tf_model->status); - TF_DeleteSession(tf_model->session, tf_model->status); - } - if (tf_model->status){ - TF_DeleteStatus(tf_model->status); - } - av_freep(&tf_model); - av_freep(&model); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + + if (tf_model->graph){ + TF_DeleteGraph(tf_model->graph); + } + if (tf_model->session){ + TF_CloseSession(tf_model->session, tf_model->status); + TF_DeleteSession(tf_model->session, tf_model->status); + } + if (tf_model->status){ + TF_DeleteStatus(tf_model->status); } + av_freep(&tf_model); + av_freep(&model); } static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) From patchwork Tue Apr 30 07:12:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48421 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323489pza; Tue, 30 Apr 2024 00:13:49 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVJERJGVclJuJvXgfOly3iJvbN5HUIyQ8lv78exF8cSPzMc3Jnrgo4So1lxOmjPHi5U1qpEGAdXBTcIb7DRHmqDzxsPyOirM3LjnA== X-Google-Smtp-Source: AGHT+IHZMLlND2FYDTT5/b7VmHcRfvM/u/VhToSo6AdClsNbjE9XiLv6Xz8bbr9nhPcjz2Bo3AgW X-Received: by 2002:a17:906:b00d:b0:a55:5ee3:3c80 with SMTP id v13-20020a170906b00d00b00a555ee33c80mr6726654ejy.29.1714461229186; Tue, 30 Apr 2024 00:13:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461229; cv=none; d=google.com; s=arc-20160816; b=aMQ6ITVI3RnKmG2TEd41+S2Z8Gd5J+zzVslK+XBmKLvz9JcPHJs2Su704fhMw+gpjR VYyD3hZHtaQs0bwSm4g9DdiCLUu3yCZOMbIeQ1Dfat25l0vzdwE3IDHUzwvNhsUXqn1R KGQsJVnXMJR9zj/7iADiuqfoOHp/v4diHzSMWhviWSCxhbfIH1iDFAzB8breBqUFhKLk U5tvwOlg62ZTae+OixNiPHUC4A2rEs7IMaB9NBaaoGbu9sWIhJx7DJ2qPEKMOzrBK0vU uyRmv10NxvNSmd3dmDdJRl6H2zTcm0uhWecUr1ZLf9XKhVvWm5YY/cCDE9eyvCMDo+fI PoMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=LlpizsvWWYaYRVDjm8+2epOCyk3Z2WFYLgjaDgZIX2Q=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=VgWzyOLjXZQxkHDfBVgxyTWqel2y/ioXkrFdtHdGv97Yt2nsB4UMCw8wy3HHmeRsZx DJ/Xxnt4VhUX5/rin2w6CU9gtFxhLrK7wIlJ6QN8ngdITvGxw3SP6noP4hwjGa7uRdQL iLGlzHP61yxgL9092JcfqUcqI5saAx0cnAk+4rRw3PU3X85jFXjMtGQHWr9UHbmBX/6X 99xDTl1S/NrX5F14DMfbyLpjeGxeYQPqFDitRqwWEyUtDil+yP+hoAKtCX3N93jKOrds uA2ioeD/B/rRep1SPPKErD4h8Fm+6TWcwAGrZszsX93oL517bosXP/umkMAtFzhk4k/0 eKcw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=OHPC0J3j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id sh32-20020a1709076ea000b00a5584804fdesi15586828ejc.378.2024.04.30.00.13.48; Tue, 30 Apr 2024 00:13:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=OHPC0J3j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4C6F968D56E; Tue, 30 Apr 2024 10:12:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-53.mail.qq.com (out203-205-251-53.mail.qq.com [203.205.251.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 44B4068D58B for ; Tue, 30 Apr 2024 10:12:23 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461135; bh=Dnif5iayuKoESi5vD/iCTlE3RzxJBVD1FT+10ZIhx2s=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=OHPC0J3j0ZGeN4DwlZWtlEyi/2XxbLL/m2j/IadhtySkWPKrFgT9+0rfkGc/Li9RT eRH4FljPpfS4P+L99CJU2lJtahpRPAOXLGv3CqEwhb8OjHNZm3F5zPnSErd2L6pWSy tBQnXLeQMR/IQwUIF6mBDfJU8qk2ab2gIUgx/bMo= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461135tvx009wvw Message-ID: X-QQ-XMAILINFO: OaubouGXmhNzXi9NKXp9qfVKb7CS1hewB1cguLVMkAeTDIZJrrb5KMx+Rx1Ums Oyoo3fZlrKWH3A96fIUhhMj6jpWEcAQHAqel554qqQkge2Opl3I03OGfKdnz9Nv0QExnjd864CCT 7fMe0ZgE/nJAnUbaTWyC0txMOYLIFCamxaDS5QIAFnU2/HdZzNFb9w68kxTXyWGbsAwXqDB1bi1q D69fiNIuhvWRRj1XZj5DKOhlpwTfPbXmbFT64vsZx22pySMgWGPG3SKNjqGaBKV41PIk/cWXPJFo lBSebQBBG4WX0AarMVC5moG9/8AFfl6AanRQH6clFkm58jLmtHeyLL67McW9yx93jfkADtert6W+ dc4jrjjbH0oXwK5sL7pSw5Ic577nqKc17TXH7Zyr2+53B67IuE62hXAKDTt1My0NSUKxXY/KUNvG fTOWOqtvcMh2sf/uffCcyLEuz6M7A93WroxFZmDY9mrmSDx1/Gh+fLomNDKhI6EqXVI8xUYLL9Yi 4hqlt34G8Rx2h71Nq+2GruzG2ZxYBpE3/+z+yskoYrB6BIBiHAHNDr/ERf4aG2hqh7oq5oycdhUR bJCTkJz8XrHDVpSusoWFOD9BAvIMh7vK+S7o1HZPB2BCvgOuzM437e1teXrTvtElpIvXntCRroaf FCHGxjOu6hF7eTxfwuVkufF3+oCIpj4c9soYVd4oKIGIFwHr0uO+1pi5u7Tk+AKij7Y1LnJBHGAs hdVi2tZiZhNtA15dAn+JXqzRKfkbhNB8roEp7En+kxF3wnTmlIKg7Xa0JYLc/zo56prMd722kqkJ LO0EGYNiJ/eYlEULzIYetOKYtuLM7SG/GflWxZrk77HN9uDP3px1Agc5POpqXzNOQ0X3DoylKKHo tBP/3VvHQzL4vWYeRbt5P5cczsQBg0BQ8H7s08dpwS+OuXS+0B8aXtc6D1MB/ZinaUULNLt24bTJ oHBCl5WLAjVXN34IBNdkAd/q5ooEbpumQ4/SZ+BcWO/CeDa3O+kOZdEnGR6zKdP5NQrXsjDRrrr3 u4Dw5hmuW8tiyXXfFL X-QQ-XMRINFO: NS+P29fieYNw95Bth2bWPxk= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:03 +0800 X-OQ-MSGID: <20240430071208.126817-6-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 05/10] avfilter/dnn_backend_tf: Fix free context at random place X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: II8F4/aRjKML From: Zhao Zhili It will be freed again by ff_dnn_uninit. --- libavfilter/dnn/dnn_backend_tf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 60f9e57fb7..3b4de6d13f 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -804,7 +804,7 @@ err: if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { destroy_request_item(&request); } - dnn_free_model_tf(&tf_model->model); + return ret; } From patchwork Tue Apr 30 07:12:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48425 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp322931pza; Tue, 30 Apr 2024 00:13:17 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVp9mb36214Sd7d7aCviTYxWokHJu3l1T8XDq4PL03UKxv47S3T4N94IX6vExYCQ75wQpZofvp9+T67BIYb8oTcaeOn+YPupjbHmg== X-Google-Smtp-Source: AGHT+IEa+mXi/KFgKgYvRX9DulrjpwHBH/88QcN0Js46KQaD+U7f5HHymB1Bq1fWZUW+YDWl7xoB X-Received: by 2002:a17:906:2993:b0:a55:b810:8678 with SMTP id x19-20020a170906299300b00a55b8108678mr11461005eje.23.1714461197298; Tue, 30 Apr 2024 00:13:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461197; cv=none; d=google.com; s=arc-20160816; b=k+A0wHpVTU+hELJd+1IR1HQrv1DueIIrY5lkubBukOMrzoLzP0C070mcmt3Z+GTwEk AwvoOTSnH2iX2BM3xMkpXjYUAaiW7yHbAZpIAMHK98vy/1Ydt0zXWyPBW1nvcsUlD2Ix f+/fGLtlNPQc7Yo/R9jOCDCet96Zv2IMVrEfnZn/ByXwmZqlFu7tK9FICe63I9IAntZ1 j54o4IR68qMDASpVizfmzXcASiK+tL0RkbW2kbZiMwQN7bVQy2hoagBNqAZshCBBEEzt 4uePJhRe/sQ8VcaC2WMBZ687UBC8jlz2vbuJoY6AG3GqkYCZlNm5p4kjwYtqnka3LaKg /NVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=qw7VzAp/hTieMGJntDLmrgnHKyE7G0rdV5FEBhIhEPo=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=FoQXNDgUWzUQAj5das/ea1ytrmzW0YPIfGR3P2OGlqaWC6mE3LNPsXTDAQlsUIvR8i cSbLhqJ7W++mpYSCAs9EHrniWbgcMZfHGN732LmGxkifm2rrd7s6b8RQBZI22kOuPYnC Ojf2s06h4tRm0+SawHlKsj5GUp02AqcPGDrga+Du7HIxE5UyzoW1vYyYmrJvPiTj8cut rMed+8kJ0vRO5xsFJGgP3C3q9EI+PiC/KGTf4CVX6iG0kWkyYRhVHhDpnn2pUz8h4YHR 3N5DzprV/y2bhs8zcG1PdLnpDyzboGfTshZfrjqEAzl3FPdCN/+LjhR4L3gWd4EJjdkG sYgw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=WdopVTPY; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q11-20020a170906b28b00b00a5887c8d50asi8043716ejz.836.2024.04.30.00.13.16; Tue, 30 Apr 2024 00:13:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=WdopVTPY; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 795C368D5E1; Tue, 30 Apr 2024 10:12:38 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7EADC68D5AA for ; Tue, 30 Apr 2024 10:12:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461136; bh=PxeaxHxaSPHlb8fmcLD367j4TZd/T8OvN3kDxoDx0Ac=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WdopVTPYvmtZ7fz8F4VogApcV+XIz3Blrf3o9kAqA4CN/ZMIrq+XgSmfor6t3cpCu cH4M5VSldCZeEF5LIsUGIgycAuIViIAI1mNw3xBpSnQRo7pwWX/tKH7NhmUU+O9/Dx AcYgcVKb36dSKJA0M4J5IeosedgJSTIa1BUbT8rA= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461135tvi3gf257 Message-ID: X-QQ-XMAILINFO: OKKHiI6c9SH3vxT3aH57sf3UKdkJGmuzfJHnaymo5u/rj7xkVCPrSYYtuHNPg3 QOf/nKwXqw9nCVXlXHUFrDZSz8YMsGvcQjwbN9tBCNpQZ+mAyoaZt7pvLtOKYVmeQ6ZG1s+QF2Je E9f+vs9kW8G9mNsWJtENVQ5KRz/R0wJBEqpxXYkk78EM0ybfXVP8vibqo6zF6lOUzU4hjoe/FI3f QUY57bh/zwQK4FxgqPHV21EPY1ZLGfFz1n6jKn7jMjJUtw2iw5VU/cAk2dWdvoStKpSgtlXzGrwf 7h/mfKaEgzYdHzqf86ji4a44QDZlJ50fCnAsm4XxvQWorNN9iXrJbyBs5xVZADhcKXg/1T19bYkZ 8obPweS9VtPEZCyrjeuT++P2argI5QVDAA5mQs0xPgNfJ1WQFGWNm+tugZd3tv5odcws6D6lBM0y 9WhEg3mZW9Jqk5oqo6FzSBL+fNWojSYR0tzQgeu4biKZ07VosYlUKUgDUjIiD8w2ij24/MsFEhFB ss5cTicE6VkuyYV9JOSbET9xB18uPFDpFiiLBaHkcxLOx1/ooUVmgVTu9kLRAHsByZ+PZP5h8ke9 NnMpvB875Y0D6GNbbAzQ3CtBPJx9Jj9WQUuU58Ke8T0n0yvQMgSktIOV9A5jftJYYd24wxS27J85 zbbx4VmEL1yk61OLAxcRlcNjEidBtwz55CaLO2bDhQQIyIFkmqyr1wCPKx9sdUHgJIPGrbNFgrlS fUuZzIKaAFHyVFzs816+MfOrVRb5VboO5MDGQfKsyiqF88Nj+1Y32+HUms4C+GJS8mm3SJhq/jXc HujfIG/lH0qDSR+kOaaGWSp+Ryjmg2Y5OdDQHweyJ+hcI7oSSUx0crBg2I1iuVySN0O6x8x5lVgK YP1xkpzJUoDWBTZym+Jk5Cg1dOzWguDwjJZWJc4xfxEvHRzEbnXXSIdeBi9EtcmcWEGjFw6Zyld1 V6Tny+JBkSHYRBiag4sTwR9ZIep9WzSf6nPhkujpR0RS2NrH6bLe1oYzjK28xDeFSakKBzaFV19I VULcsIHc/dH0awQkco X-QQ-XMRINFO: MPJ6Tf5t3I/ycC2BItcBVIA= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:04 +0800 X-OQ-MSGID: <20240430071208.126817-7-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 06/10] avfilter/dnn_backend_tf: Simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Xk5SjyucobnB From: Zhao Zhili --- libavfilter/dnn/dnn_backend_tf.c | 33 +++++++++++++------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 3b4de6d13f..c7716e696d 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -37,8 +37,8 @@ #include typedef struct TFModel { + DNNModel model; DnnContext *ctx; - DNNModel *model; TF_Graph *graph; TF_Session *session; TF_Status *status; @@ -518,7 +518,7 @@ static void dnn_free_model_tf(DNNModel **model) TF_DeleteStatus(tf_model->status); } av_freep(&tf_model); - av_freep(&model); + *model = NULL; } static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) @@ -526,18 +526,11 @@ static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, A DNNModel *model = NULL; TFModel *tf_model = NULL; - model = av_mallocz(sizeof(DNNModel)); - if (!model){ - return NULL; - } - tf_model = av_mallocz(sizeof(TFModel)); - if (!tf_model){ - av_freep(&model); + if (!tf_model) return NULL; - } + model = &tf_model->model; model->model = tf_model; - tf_model->model = model; tf_model->ctx = ctx; if (load_tf_model(tf_model, ctx->model_filename) != 0){ @@ -650,11 +643,11 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { } input.data = (float *)TF_TensorData(infer_request->input_tensor); - switch (tf_model->model->func_type) { + switch (tf_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (tf_model->model->frame_pre_proc != NULL) { - tf_model->model->frame_pre_proc(task->in_frame, &input, tf_model->model->filter_ctx); + if (tf_model->model.frame_pre_proc != NULL) { + tf_model->model.frame_pre_proc(task->in_frame, &input, tf_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } @@ -664,7 +657,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { ff_frame_to_dnn_detect(task->in_frame, &input, ctx); break; default: - avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model->func_type); + avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model.func_type); break; } @@ -724,12 +717,12 @@ static void infer_completion_callback(void *args) { outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); outputs[i].dt = (DNNDataType)TF_TensorType(infer_request->output_tensors[i]); } - switch (tf_model->model->func_type) { + switch (tf_model->model.func_type) { case DFT_PROCESS_FRAME: //it only support 1 output if it's frame in & frame out if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + if (tf_model->model.frame_post_proc != NULL) { + tf_model->model.frame_post_proc(task->out_frame, outputs, tf_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } @@ -741,11 +734,11 @@ static void infer_completion_callback(void *args) { } break; case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { + if (!tf_model->model.detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return; } - tf_model->model->detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + tf_model->model.detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model.filter_ctx); break; default: av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); From patchwork Tue Apr 30 07:12:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48428 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323652pza; Tue, 30 Apr 2024 00:13:59 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV0mIEQJnt7wiqM8MjYoHUgsOdn3EW7cVUYas2lwOS5r/BKVbEK2cfujSOeZyhQZvNCSK27P88MTLgln6u3VsP3EO1Cqt/z1WHrLQ== X-Google-Smtp-Source: AGHT+IHj4deIXzvfpxONbhdynxOjO538vgySs7t6si67CJuruZ2v2eEk6jqNr8X45yqFE2oSN3La X-Received: by 2002:a17:906:c349:b0:a52:5b71:d91f with SMTP id ci9-20020a170906c34900b00a525b71d91fmr6190171ejb.19.1714461238870; Tue, 30 Apr 2024 00:13:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461238; cv=none; d=google.com; s=arc-20160816; b=Ee+tENbtTidhpi2v5jZSosd0SJWIRNjhlfUXcVBGL/i7+feS5QqjMHdggq/qVDGGeu /HCoseN+Fw77G3X5Ui+Om0KxqZIkPvHrh4i4/6N1lJ5Y/G4bS+cIxjNvh1VAhxrjQhrt pPR8LkGS/AlO2+Y/vglOTkg07r2NGcJpDHuj3oSev8eCYgPtyc+QYXyoPaLzlYKZx/LO Fuyf8+U7pd3Q+SbRf8SscSPUPgfbOIuHj/Wf6m8nfr3aooZ7rLFe7b5LzdUufpBEiFZv /hRFzSBKm1rfPJrcakJPicJXqtKzM/dehy2TMhMjNIOVV5ab2w+yGaPuNIzsQrsFUvS2 m/Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=6YKkislD9HrmFXCzt+kV4QkjjuLxomhieYfWvKnx0cI=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=ZlOwkymCzIAYBbI2ACE5mNmV+30v+CxLDeA7NjDSK/2IiLU4K6zaxtYlKCEA4r+/b6 4RxNK9oDt1DwRc/P/0A6FlaPXKMsxWQWAsmoCnk1kBont/gLtSP8RxDzUDhaUYdi5pPN VbmRME56a3vP1y02PUOqgDohHCt5jM2wT9DcoMTuM0OeKyXdJLvURyy9xWgd9GnHp3zH 3AM7stXomzOnnfkNn3doGSh9Idwproz/xv3wzlGUaSXsXTYNKwDaNepxxIPjeYRRoWcW 1f0qgFioz+DGgPwqo+N3ZLwz5Fse0QerrsmoQq3vB+CiU6bxDOEySkAMLCEtgXqUtseO AG4w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=wu1bGG3j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z9-20020a1709063ac900b00a4743988f95si14552813ejd.820.2024.04.30.00.13.58; Tue, 30 Apr 2024 00:13:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=wu1bGG3j; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E128668D5D7; Tue, 30 Apr 2024 10:12:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-252.mail.qq.com (out162-62-57-252.mail.qq.com [162.62.57.252]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CA2D168D5B6 for ; Tue, 30 Apr 2024 10:12:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461136; bh=t8GRrPziAy/DBR0es5Cliii90QdzrzGP9RaqvJDGK/M=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=wu1bGG3jfz4+xk3EpjlX0cTKvPy9jNkEwelHuNtfT84ta0l2sX2BVrD5pzC9cfkbh YXxDTSOHDau8Mx8gk58O2wVCuhiJqrUNV07rkEyf8XuLUlBha1hxiOwBeIBdJKHd3n CEu+FbOO15RwzEtFdKKU2R5o6CdHFW3qZcrYoDTQ= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461136tm199b3rb Message-ID: X-QQ-XMAILINFO: N9lasXYeZMXWZB8t8XXK42e5JSO0nIONeOiQBkeWvxbl+qtMV5202y3HY2i3QP CO8InYXa4/zfSRcRsufIp92hfWGC34c3twlisWF/ybNQpFxWSJZe1lj9KhB6QVUDth4VeHA7Ilxg lA5jvfpL6icXVTfA672nMb/SymNosNBeUd7ejWSXSA8mdNsKzkOqt/dkwBV0Z4asrutYXqpYrP52 17qyuRf7eeeUE0nOIpIlf/TNYHMab7szjxPF7tHtqnWGC99e8ioNtbPl3tqw3ts6u6O7kA025QIp pwqoyJN9oyNpsBeCSdjxSiErrVlGvDc1vEwY36Z15304P1gNZb+8fEht4aDnpB8bdCXwxKol8SXI Vf4JtfXzXllDMIXdwWytO7sC6DgC2xZa/zuqaejMj1hVjiH8erW3QV4RZzpvtU+U1L9Lz/NIUv48 Ag3fOtpDu/UHdGZJ3UHULHml6R/54n1SP6+9epokvUgQlAiuCb76Cv1EVXI/ElZ+1IfxIYT9Pupn /5AHbfvhBdfCUAiaHxkc7t7Ou3SbmuuJAzrRgDE4HxqYwFtMpwKM/uzn5745XHaJERkVhE8Kbyj6 OB1I4gUyZPb4En8MMOh6SaX1a5Uxm9TQRd6i/SD4gAkx/FSFRcLlqZwy7hVLsE6ssGc1ixIB+E1k 4F9bJOyZJp+qD3RLDjnPYkZ5SeEVWgfUtMizvt3jAO/YoM9/ndpADp7OvLHF3iSJn68AY1km/sYy j7rvYLvphyMfaAV+TVH/OSqGxso3uBjK7Y78TzV5JEgYlPopOPNE4crLofg+l2MZvRm9JmC33cIw RIP2568wO5WQhYPnR9JNgTnBwqK46hSFDC24uvsZOP2LfJ5cqz7CVXJ2av6yloXzkmB8OJ7qTlir MKEbL3uAgtQkVCc9Ugl1qGbGCTSTvUeuSuTDqwiGx9lNlRIe3yJcigBBp6YnZHV2rvd9DNy61ILN D3Ae779bg0lbco9uHMyTvKt0ZSW5AdRCKDc4wCXX5OhfgdA+6rC4qCm7REq/1K2Lo4vVxl95sklz gf/IpbsQlEPrORbbw0S9Wvh7zCiDw= X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:05 +0800 X-OQ-MSGID: <20240430071208.126817-8-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 07/10] avfilter/dnn_backend_torch: Simplify memory allocation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: SBsMj2k5SK8F From: Zhao Zhili --- libavfilter/dnn/dnn_backend_torch.cpp | 31 +++++++++++---------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index abdef1f178..818ec5b713 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -37,8 +37,8 @@ extern "C" { } typedef struct THModel { + DNNModel model; DnnContext *ctx; - DNNModel *model; torch::jit::Module *jit_model; SafeQueue *request_queue; Queue *task_queue; @@ -141,7 +141,7 @@ static void dnn_free_model_th(DNNModel **model) ff_queue_destroy(th_model->task_queue); delete th_model->jit_model; av_freep(&th_model); - av_freep(model); + *model = NULL; } static int get_input_th(void *model, DNNData *input, const char *input_name) @@ -195,19 +195,19 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) infer_request->input_tensor = new torch::Tensor(); infer_request->output = new torch::Tensor(); - switch (th_model->model->func_type) { + switch (th_model->model.func_type) { case DFT_PROCESS_FRAME: input.scale = 255; if (task->do_ioproc) { - if (th_model->model->frame_pre_proc != NULL) { - th_model->model->frame_pre_proc(task->in_frame, &input, th_model->model->filter_ctx); + if (th_model->model.frame_pre_proc != NULL) { + th_model->model.frame_pre_proc(task->in_frame, &input, th_model->model.filter_ctx); } else { ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } } break; default: - avpriv_report_missing_feature(NULL, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(NULL, "model function type %d", th_model->model.func_type); break; } *infer_request->input_tensor = torch::from_blob(input.data, @@ -282,13 +282,13 @@ static void infer_completion_callback(void *args) { goto err; } - switch (th_model->model->func_type) { + switch (th_model->model.func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { outputs.scale = 255; outputs.data = output->data_ptr(); - if (th_model->model->frame_post_proc != NULL) { - th_model->model->frame_post_proc(task->out_frame, &outputs, th_model->model->filter_ctx); + if (th_model->model.frame_post_proc != NULL) { + th_model->model.frame_post_proc(task->out_frame, &outputs, th_model->model.filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, &outputs, th_model->ctx); } @@ -298,7 +298,7 @@ static void infer_completion_callback(void *args) { } break; default: - avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model->func_type); + avpriv_report_missing_feature(th_model->ctx, "model function type %d", th_model->model.func_type); goto err; } task->inference_done++; @@ -417,17 +417,10 @@ static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, A THRequestItem *item = NULL; const char *device_name = ctx->device ? ctx->device : "cpu"; - model = (DNNModel *)av_mallocz(sizeof(DNNModel)); - if (!model) { - return NULL; - } - th_model = (THModel *)av_mallocz(sizeof(THModel)); - if (!th_model) { - av_freep(&model); + if (!th_model) return NULL; - } - th_model->model = model; + model = &th_model->model; model->model = th_model; th_model->ctx = ctx; From patchwork Tue Apr 30 07:12:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48427 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323313pza; Tue, 30 Apr 2024 00:13:39 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWznWuJHwAw0lUHkHDiLFHqK6AAt6hP9UPFudEnXPg8S3KRbQ5Rt+08ugzcmjIDsKu06DYYHE/jcphYGh73x//EaYi+c88czvVsAA== X-Google-Smtp-Source: AGHT+IHo9tn6b7cEFnG7nk651MkUVJHWM/KjcjcvavzaJBWNm2CGx2RJSy+54erlwdKzu45sm83H X-Received: by 2002:a17:906:140a:b0:a55:b05b:cdf2 with SMTP id p10-20020a170906140a00b00a55b05bcdf2mr1348628ejc.21.1714461219263; Tue, 30 Apr 2024 00:13:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461219; cv=none; d=google.com; s=arc-20160816; b=F4u15PsqaKaMnaB17UZI7SzF1b7hq3mPdJVeh8T1M17gUChq6pebcJCfSL/AKj9PUs NjBxicQTCi9iUc6up8bDJ0OMJvFIkyGkk+8Zk4cFh1rmTZ1ddEYe7kftt7wj0YusCrPJ JiHxAiDZ0LJWPxWoksaMEkKsBk9h2u1igyD9nWB9nfzu6+AFugaHAmr1pKinhCu73wHe +OofJAC27VcpOtXzN8FyMUK5oSivgkrV5M2Dy/850NXun8Cag0FLZywq04ROoaOE6BgJ e6DOakcNqemKF2EXuM9aiT88GyS/yR6PFp1iyCinGQarUhwehmCA1OfBds+c84Pxk6Hm TMSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=vWi3GoSc/D7KTNuzJyyYKML+Gchy5xRQ/lEhd2xRYGg=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=zHoqf26vdCjh9V6Ey5uSTU70BYLJ8TM+giz2Rx62NgoaIvbCSVz01eRnfGf0DaYO7Z jt/gLErfFO4l6qZ+udH2q18Vdg+1npETkXUGm1v79D+NUO9N/Pz5o+pX3oRx95zu577K 2ZO5bdC9YR0AxSFkgnZ4xoa7mUyvXBJlYN8TEQFYesWqDwLeJw66z2k07ngaFtR0LnbS B81cy/NfVBlYHtQPfi0AGiqjk/tZZevHzcsS0UM/1ZzhPF2TN6I614htLLNry4XyJt2l IxUYaTOdqq8WNm4FAG8tJ2UN6uDkCP5Lg1hN2mFffZBKZWtWdtBkN9FUUZRrKvoPK/qM iRAw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=giqCRDOU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ga39-20020a1709070c2700b00a55be3c65dcsi10875532ejc.238.2024.04.30.00.13.38; Tue, 30 Apr 2024 00:13:39 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=giqCRDOU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 73ED868D555; Tue, 30 Apr 2024 10:12:41 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-252.mail.qq.com (out162-62-57-252.mail.qq.com [162.62.57.252]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9DD3B68D5AE for ; Tue, 30 Apr 2024 10:12:24 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461137; bh=GhKaAHSv7wf4qcgN1IDpIdWTbP5243HVTDilGfgWKyE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=giqCRDOUaKuNqHFRiTzf5yaV1RuWkw42g/oro6NtxfIuvj3aHDfhhT5Ib0joc9THg Pq9HSk3wQPBSarso7nBoLz5Lwexviu2bkxAdGnbeCS0POINh40/A781Eu2hbC/XnI5 2jkgJarjRCXBKmTOaYAXoYgcKw+FhAbj4EQqssm4= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461136tohmh9uc2 Message-ID: X-QQ-XMAILINFO: NvfE96cLltb5gWlwT6ROpiAAWp3Y/oYH7p3Slxl9OvRFspWMKiO25r3WCHfN79 jToESDOj9XHbycLQYGulsWD6TlSmyhdY8JdOalgcHODf72YGYgrF2mGUywck8Zw8tDVIpbr5pchS TUj8EaQpCXMw8tljORO9rZUdK526L3GtG9efeZxf5FJxfl6BwxismrqEGM0Y9Tjp+CHiWHJkJNyf l8gbOIX9CLEUXUgDW43nwJ+wZcO7wkEmF1R2sRyfGArrCnuqhXl6lnQR3KBJtaZxz8HDF8E6aA1M rtmTQiN9dEBmtEQWt/nDbQO7JXrvyqCWnaamQ6uaxFfSw62CUOhvlwPzaFPUrwdTBX5/R12W/sVr /J9RML1RxXZwoC3AvkOFRvnc2xRETy29N1jX0j543LXCljd0q3iKbZponGNNExuZjIVzHz5dS+BN 4oc5D2e0zKzOby5M9R56cxiKXK04mri49IMEU9Bk+BeIF4ZJNib0hN2k/h1nmBB6Gn8jqWNuCgK7 VZUqy0eBLV93LkTv6PQQ4UOaZqTvy/botG9W+jD32xC5ymyg8q8ubTfrriPqyGd91hAfOUzUrddy B51wB77o+jQgYpqtR7HiHHnL911qBzwUbBF+xSevVSb7RWtogoyodXy4HZmBoDJlIGWP4Vmwtjv0 yfqHYF6BNrK6009PYzXnDQNPUjHhLQg28xjjU04E75M99azkXPFUxj8Wuuw3HgVPSY2Kz+f+hQGu Tln4pTG2kE6wIqnJe57D6TqS6cRFgbyY/TzrDTbG/qNeS2wICGon5KgCbXWHODRatFSrK3D+V0/x D/cvKYTQSNdqk1LvxG8kQqxkdZQ97LLsq7hZ+xr3VmRqAsFTLwfjX9GMKihNFJ281ss4RRW8bQm5 T2QoZEQIwyDz2G1xxH2axCoiYJ507UEEErTX0fUfuxgcEiBIgBAFQDk2ka07RZyurexWU8JreDpb 9+6YPfgC3yKWdMYmZ8S6TGWFisY35WhhTaxt2edMQwqAn7I4ioJS1sjXVCS2yTjnCmxhDd7dUzZr BQaF6hex08C31M04W0YTCmEh9lt1h7lr6PdeHYbgROIzpz/a0uKxZ0QYfxQk4= X-QQ-XMRINFO: Mp0Kj//9VHAxr69bL5MkOOs= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:06 +0800 X-OQ-MSGID: <20240430071208.126817-9-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 08/10] avfilter/dnn: Remove a level of dereference X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: LWZbhXUa+iDh From: Zhao Zhili For code such as 'model->model = ov_model' is confusing. We can just drop the member variable and use cast to get the subclass. --- libavfilter/dnn/dnn_backend_openvino.c | 17 ++++++++--------- libavfilter/dnn/dnn_backend_tf.c | 19 +++++++++---------- libavfilter/dnn/dnn_backend_torch.cpp | 15 +++++++-------- libavfilter/dnn_filter_common.c | 6 +++--- libavfilter/dnn_interface.h | 6 ++---- 5 files changed, 29 insertions(+), 34 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 1acc54b791..d8a6820dc2 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -517,7 +517,7 @@ static void dnn_free_model_ov(DNNModel **model) if (!model || !*model) return; - ov_model = (*model)->model; + ov_model = (OVModel *)(*model); while (ff_safe_queue_size(ov_model->request_queue) != 0) { OVRequestItem *item = ff_safe_queue_pop_front(ov_model->request_queue); if (item && item->infer_request) { @@ -1059,9 +1059,9 @@ err: return ret; } -static int get_input_ov(void *model, DNNData *input, const char *input_name) +static int get_input_ov(DNNModel *model, DNNData *input, const char *input_name) { - OVModel *ov_model = model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; int input_resizable = ctx->ov_option.input_resizable; @@ -1255,7 +1255,7 @@ static int extract_lltask_from_task(DNNFunctionType func_type, TaskItem *task, Q } } -static int get_output_ov(void *model, const char *input_name, int input_width, int input_height, +static int get_output_ov(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { #if HAVE_OPENVINO2 @@ -1268,7 +1268,7 @@ static int get_output_ov(void *model, const char *input_name, int input_width, i input_shapes_t input_shapes; #endif int ret; - OVModel *ov_model = model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; TaskItem task; OVRequestItem *request; @@ -1383,7 +1383,6 @@ static DNNModel *dnn_load_model_ov(DnnContext *ctx, DNNFunctionType func_type, A return NULL; ov_model->ctx = ctx; model = &ov_model->model; - model->model = ov_model; #if HAVE_OPENVINO2 status = ov_core_create(&core); @@ -1470,7 +1469,7 @@ err: static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_params) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; OVRequestItem *request; TaskItem *task; @@ -1558,13 +1557,13 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; return ff_dnn_get_result_common(ov_model->task_queue, in, out); } static int dnn_flush_ov(const DNNModel *model) { - OVModel *ov_model = model->model; + OVModel *ov_model = (OVModel *)model; DnnContext *ctx = ov_model->ctx; OVRequestItem *request; #if HAVE_OPENVINO2 diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index c7716e696d..06ea6cbb8c 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -262,9 +262,9 @@ static TF_Tensor *allocate_input_tensor(const DNNData *input) input_dims[1] * input_dims[2] * input_dims[3] * size); } -static int get_input_tf(void *model, DNNData *input, const char *input_name) +static int get_input_tf(DNNModel *model, DNNData *input, const char *input_name) { - TFModel *tf_model = model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TF_Status *status; TF_DataType dt; @@ -310,11 +310,11 @@ static int get_input_tf(void *model, DNNData *input, const char *input_name) return 0; } -static int get_output_tf(void *model, const char *input_name, int input_width, int input_height, +static int get_output_tf(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { int ret; - TFModel *tf_model = model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TaskItem task; TFRequestItem *request; @@ -486,7 +486,7 @@ static void dnn_free_model_tf(DNNModel **model) if (!model || !*model) return; - tf_model = (*model)->model; + tf_model = (TFModel *)(*model); while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); destroy_request_item(&item); @@ -530,7 +530,6 @@ static DNNModel *dnn_load_model_tf(DnnContext *ctx, DNNFunctionType func_type, A if (!tf_model) return NULL; model = &tf_model->model; - model->model = tf_model; tf_model->ctx = ctx; if (load_tf_model(tf_model, ctx->model_filename) != 0){ @@ -611,7 +610,7 @@ static int fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { task = lltask->task; request->lltask = lltask; - ret = get_input_tf(tf_model, &input, task->input_name); + ret = get_input_tf(&tf_model->model, &input, task->input_name); if (ret != 0) { goto err; } @@ -803,7 +802,7 @@ err: static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TaskItem *task; TFRequestItem *request; @@ -851,13 +850,13 @@ static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; return ff_dnn_get_result_common(tf_model->task_queue, in, out); } static int dnn_flush_tf(const DNNModel *model) { - TFModel *tf_model = model->model; + TFModel *tf_model = (TFModel *)model; DnnContext *ctx = tf_model->ctx; TFRequestItem *request; int ret; diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 818ec5b713..24e9f2c8e2 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -119,7 +119,7 @@ static void dnn_free_model_th(DNNModel **model) if (!model || !*model) return; - th_model = (THModel *) (*model)->model; + th_model = (THModel *) (*model); while (ff_safe_queue_size(th_model->request_queue) != 0) { THRequestItem *item = (THRequestItem *)ff_safe_queue_pop_front(th_model->request_queue); destroy_request_item(&item); @@ -144,7 +144,7 @@ static void dnn_free_model_th(DNNModel **model) *model = NULL; } -static int get_input_th(void *model, DNNData *input, const char *input_name) +static int get_input_th(DNNModel *model, DNNData *input, const char *input_name) { input->dt = DNN_FLOAT; input->order = DCO_RGB; @@ -179,7 +179,7 @@ static int fill_model_input_th(THModel *th_model, THRequestItem *request) task = lltask->task; infer_request = request->infer_request; - ret = get_input_th(th_model, &input, NULL); + ret = get_input_th(&th_model->model, &input, NULL); if ( ret != 0) { goto err; } @@ -356,7 +356,7 @@ err: return ret; } -static int get_output_th(void *model, const char *input_name, int input_width, int input_height, +static int get_output_th(DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height) { int ret = 0; @@ -421,7 +421,6 @@ static DNNModel *dnn_load_model_th(DnnContext *ctx, DNNFunctionType func_type, A if (!th_model) return NULL; model = &th_model->model; - model->model = th_model; th_model->ctx = ctx; c10::Device device = c10::Device(device_name); @@ -489,7 +488,7 @@ fail: static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_params) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; DnnContext *ctx = th_model->ctx; TaskItem *task; THRequestItem *request; @@ -538,13 +537,13 @@ static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_p static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame **in, AVFrame **out) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; return ff_dnn_get_result_common(th_model->task_queue, in, out); } static int dnn_flush_th(const DNNModel *model) { - THModel *th_model = (THModel *)model->model; + THModel *th_model = (THModel *)model; THRequestItem *request; if (ff_queue_size(th_model->lltask_queue) == 0) diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 860ca7591f..6b9c6f8d7f 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -157,15 +157,15 @@ int ff_dnn_set_classify_post_proc(DnnContext *ctx, ClassifyPostProc post_proc) int ff_dnn_get_input(DnnContext *ctx, DNNData *input) { - return ctx->model->get_input(ctx->model->model, input, ctx->model_inputname); + return ctx->model->get_input(ctx->model, input, ctx->model_inputname); } int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height) { char * output_name = ctx->model_outputnames && ctx->backend_type != DNN_TH ? ctx->model_outputnames[0] : NULL; - return ctx->model->get_output(ctx->model->model, ctx->model_inputname, input_width, input_height, - (const char *)output_name, output_width, output_height); + return ctx->model->get_output(ctx->model, ctx->model_inputname, input_width, input_height, + (const char *)output_name, output_width, output_height); } int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame) diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 4e544486cc..1154d50629 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -91,17 +91,15 @@ typedef int (*DetectPostProc)(AVFrame *frame, DNNData *output, uint32_t nb, AVFi typedef int (*ClassifyPostProc)(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx); typedef struct DNNModel{ - // Stores model that can be different for different backends. - void *model; // Stores FilterContext used for the interaction between AVFrame and DNNData AVFilterContext *filter_ctx; // Stores function type of the model DNNFunctionType func_type; // Gets model input information // Just reuse struct DNNData here, actually the DNNData.data field is not needed. - int (*get_input)(void *model, DNNData *input, const char *input_name); + int (*get_input)(struct DNNModel *model, DNNData *input, const char *input_name); // Gets model output width/height with given input w/h - int (*get_output)(void *model, const char *input_name, int input_width, int input_height, + int (*get_output)(struct DNNModel *model, const char *input_name, int input_width, int input_height, const char *output_name, int *output_width, int *output_height); // set the pre process to transfer data from AVFrame to DNNData // the default implementation within DNN is used if it is not provided by the filter From patchwork Tue Apr 30 07:12:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48429 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323781pza; Tue, 30 Apr 2024 00:14:09 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXMUfBz68ZC5I/3Q78+w0DiNSGZymm6Vyh+Nz4yT96UpAID0NFjBA9YsRraNzCpWJD78mrrHzxz+fZ8LNsvcy4KRt7Np/oG1r9S6g== X-Google-Smtp-Source: AGHT+IHow09mMFG7KFecPc2cIEtcml9Tcc+XtFxHs4mOr51qPuC1OcAcJapv5fldWSgtCM615TMx X-Received: by 2002:a05:6402:3609:b0:572:7dc7:4e36 with SMTP id el9-20020a056402360900b005727dc74e36mr4599269edb.3.1714461248705; Tue, 30 Apr 2024 00:14:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461248; cv=none; d=google.com; s=arc-20160816; b=zTgjKbNEAcb8WrUUD0tf9pU0fD8RN/Rgwn7Dp6zfnpcBU9MWAA+wFibPMOcawg9AGG JTCqRUkN7EWRWqYRmuPboIJUdDAcl7Trkcxzep4FFub4c1ArOkpnZbh3lvIxyEwPJ62x z9XJyjaF6lFb0ia5uKRr5bknVrd0NPoITqGM7dB4gJMPq2tN5r/H62D/l3NrmztgV/oR RFdhgWGnXDrIRJxT+0Rqk+8xOa7B7JM3Nsux9CVBHO0OVgvuwhYyezX2H0Mgv7qbXPRt jIhMfBUkhwq7bjxWx4RgpDi39sdVKz0Ub1/BJnB7NKQntVKUdTsaGfyrqWC9GpX45e9e GSOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=GseeTnS212FMC6lJQp4PefFR9ml8U6t4aNQfYklzVwg=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=p2l6giNwEkCGaG0WksOhLxFEnY/CrS0n+wFKkj99aAnbBYbkw/HyE1Z8XLF63srJjB uxcJoRMyNkScZPyqGWfi7TWYc22EOYWa9p+eMukrrttHJ+SGkEyu9/VQ6G3t7EqUa8J6 ANxIfHYkZtsO/MnaZpEu6FBirP+0YUHDZP3Pmw3ZZGXss5PO9gVICE/JQ73uVvePiT9b oID4NkGUXbkGTTejyCiLKIXqdT/xFH64vnxcHvGIVw5BOFwSx4n5ApwYcLWVB/CEduMl JA4klL0Jtu+EVSmnDLsdLaE7xXrdjnr8Nw7dsLQt0P+MJe6zgZDtabPDPwZ+aajsxfdh hmGQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=LX5lf8bx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f10-20020a50a6ca000000b00571d8dfdb52si13393358edc.83.2024.04.30.00.14.08; Tue, 30 Apr 2024 00:14:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=LX5lf8bx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BF64568D568; Tue, 30 Apr 2024 10:12:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-233.mail.qq.com (out203-205-221-233.mail.qq.com [203.205.221.233]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id F2D0668D541 for ; Tue, 30 Apr 2024 10:12:26 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461137; bh=0vJA8Dl1sAkmG0nGbNaA8Hl/HMqrUq7btaRHA+p6wCE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LX5lf8bxJ1MdpFwsudjIGiwleGL1rRbGshMJTuKxxN/5oMMUe7GsPAZPdznlgi8E1 qrHfdQ5nyho5bCSVCN8NsURFpq4YSkY0OdjpTBdvFwtKmawU6Nccj9KYuCQfnCyOKS EGYFZsF3nrKjR3womuP00e0BFUHh90kHbYKJUH3w= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461137tqo3mn7m8 Message-ID: X-QQ-XMAILINFO: MyIXMys/8kCt65l09izoTVPkM4rG8U0fE+lJWn8xPTuodhAfagt7cxTXP7jg8J kpIlfV77C03TQbsBjE7MRJZtVfZCsiG/zzEK/HbiLMxgYhD6JUvU9h7cDAcD3tXL0V0frspffoKe Uv6Hjv84tx5ZnGFKSsKIro0oHlKmk7lZWsAFs0R3mRtfA0JpYSAYUbLaD+zOvAT+W2uY2r9CQRKH 7wySaX73qkfKtJz2kuzyono7VaeoJtQxOGa0DwI7LrxQY/JhXdnbgBCL2d4dAvOIQmHlNqXwbSff tzBMAE2BdI0+4/cbFQn2H5rp9CPS2HpPBYk/RjMx8qmLyifFuSTwBXLWc5b/Zu93Tt07obI5LMIS B5AFj+n5pl3oUzN+10UdluRPsThBN5VrPpalNw4c4IUk5Icv5UeBGvfXdbRz5Qcin+OW9FAGfZSv yIt3hAv3aVsnm0spi8g+rS0WpeM5xW2Y0HFnvzWifApzexSy9YNdBRI1txp1WZAFdJNIMwJTXCfh u9IP4LFegz+ir/F62d1PNY62bLcL1mRUHo3PKvzFVCaVXH3mu3Qf79tSdLnIi4vZgaureG1iFDQk xha2xsT37GliNHl4uZjkZe8JWWsryEoz8JEHRutdNJwSDKwYHx4dHQV/NhCD/Qakq5Hk0wmV621S /Z0kfhEd766qmD35C1UGyWKyIqzmoMOOCtTEWypICe0S2eJb0PmgLg6Y57ShklohCt89S/UGRdDp UM4rWGiLkuuvypRMCyvkvqwn6AZt3/Hj9IOl5+e1SfgjICTpvxsxMSUW/aF0YLx9c9WgzSfPWrGK MvyRTe/BJduqHGQA9Xiby/Ly4vGC06LPDU6xvgAJvvcz/4KGPUPOc7wc421jvBVZvEoRmrsgXlsY 1PJ4yYjk/dXKwjjVh+RYz9FuNPG3deUm2K4W2YMbWWgT4CUaA5sVeNRywKAWnIeyETpOZQq7XPME dAmTvYSD6aWVpA4eHj2fViizBZA7Cr8dZUWCskoamIFi7yuiJHQWg2wCvREOuj2sW1mKBlIMKP0r /zEopqCd7C8obM1YS0 X-QQ-XMRINFO: OWPUhxQsoeAVDbp3OJHYyFg= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:07 +0800 X-OQ-MSGID: <20240430071208.126817-10-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 09/10] avfilter/dnn: Use dnn_backend_info_list to search for dnn module X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: wqjJAz60aNPC From: Zhao Zhili --- libavfilter/dnn/dnn_backend_openvino.c | 1 + libavfilter/dnn/dnn_backend_tf.c | 1 + libavfilter/dnn/dnn_backend_torch.cpp | 1 + libavfilter/dnn/dnn_interface.c | 26 ++++++++------------------ libavfilter/dnn_interface.h | 1 + 5 files changed, 12 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index d8a6820dc2..9c699cdc8c 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -1613,6 +1613,7 @@ static int dnn_flush_ov(const DNNModel *model) const DNNModule ff_dnn_backend_openvino = { .clazz = DNN_DEFINE_CLASS(dnn_openvino), + .type = DNN_OV, .load_model = dnn_load_model_ov, .execute_model = dnn_execute_model_ov, .get_result = dnn_get_result_ov, diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 06ea6cbb8c..6afefe8115 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -886,6 +886,7 @@ static int dnn_flush_tf(const DNNModel *model) const DNNModule ff_dnn_backend_tf = { .clazz = DNN_DEFINE_CLASS(dnn_tensorflow), + .type = DNN_TF, .load_model = dnn_load_model_tf, .execute_model = dnn_execute_model_tf, .get_result = dnn_get_result_tf, diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 24e9f2c8e2..2557264713 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -561,6 +561,7 @@ static int dnn_flush_th(const DNNModel *model) extern const DNNModule ff_dnn_backend_torch = { .clazz = DNN_DEFINE_CLASS(dnn_th), + .type = DNN_TH, .load_model = dnn_load_model_th, .execute_model = dnn_execute_model_th, .get_result = dnn_get_result_th, diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 2d3bd229a4..b56c22e4c7 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -81,25 +81,15 @@ static const DnnBackendInfo dnn_backend_info_list[] = { const DNNModule *ff_get_dnn_module(DNNBackendType backend_type, void *log_ctx) { - switch(backend_type){ - #if (CONFIG_LIBTENSORFLOW == 1) - case DNN_TF: - return &ff_dnn_backend_tf; - #endif - #if (CONFIG_LIBOPENVINO == 1) - case DNN_OV: - return &ff_dnn_backend_openvino; - #endif - #if (CONFIG_LIBTORCH == 1) - case DNN_TH: - return &ff_dnn_backend_torch; - #endif - default: - av_log(log_ctx, AV_LOG_ERROR, - "Module backend_type %d is not supported or enabled.\n", - backend_type); - return NULL; + for (int i = 1; i < FF_ARRAY_ELEMS(dnn_backend_info_list); i++) { + if (dnn_backend_info_list[i].module->type == backend_type) + return dnn_backend_info_list[i].module; } + + av_log(log_ctx, AV_LOG_ERROR, + "Module backend_type %d is not supported or enabled.\n", + backend_type); + return NULL; } void ff_dnn_init_child_class(DnnContext *ctx) diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 1154d50629..e2582c808e 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -170,6 +170,7 @@ typedef struct DnnContext { // Stores pointers to functions for loading, executing, freeing DNN models for one of the backends. struct DNNModule { const AVClass clazz; + DNNBackendType type; // Loads model and parameters from given file. Returns NULL if it is not possible. DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); // Executes model with specified input and output. Returns the error code otherwise. From patchwork Tue Apr 30 07:12:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 48430 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5007:b0:1a9:af23:56c1 with SMTP id n7csp323930pza; Tue, 30 Apr 2024 00:14:18 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWwc/U3vkcY+5G85sk2AonEP5MCwJgU7tl1JFoohJCNYcXbL8mqcAB1CXr2D1tfb6ps0x8VP2vn3+/LiXb7uKvYVcKuh3mz6bGRwg== X-Google-Smtp-Source: AGHT+IG+a7tYmc0hkMOL2/GxAXXnv6z3qZ1LUDMvzcVxCQn5F62wb5mHRAuHJvdZYF0U6wjeqrn2 X-Received: by 2002:a17:906:e59:b0:a58:921a:970d with SMTP id q25-20020a1709060e5900b00a58921a970dmr1669883eji.6.1714461258019; Tue, 30 Apr 2024 00:14:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1714461258; cv=none; d=google.com; s=arc-20160816; b=wGM4gN8mvtmwWr0SKiC6L16masNcgucEOqjgC6u+fBU7y5HfRZhbOAmIAqKILwowor dEVyrFYCsuwmt6BZiATldwF4ObDcZTBHLAadKgOBZ3CKNwmXVpvxh9wyEgDC0qvcV300 zf3iYfK4IghbCvW4F+eB6kWz/6uZLt95JjhMPVKYdMwgcG9+V3fkUDw3CqfIFxQdUldp 5oH5FN25ov3IKdzZw0u3n94t6G0tLVpmxOLtlqwgbKJ15+mUfpTgf7arTG6ZSBHJqakp Tzw9IK5DH0TERC7dZV4mpVqMsAtTi2rHh4qDe/PE2KY648QJ73ApO2vWnAuHcYk3+9jX FAkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=bNJzmvp/kAe1KqzQ4x8i2wmo6bC16+o+oMyJt7SacXY=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=LViYtAQq7Y2Rypd8NM1/mdB+Hvq/a+pI+1tRqg2AngskUsq5gZiFENRW2HhDx3fD5Q JkzjXZca8Kf+k8nvVz5a/qZ9MTTtBix0bwk3L22uWkDVq0hZk81aL6Q8XQiJiTZQpHJ7 dOKIaDpbDQHluV+rfvXOBsXK4wLmNeBj5w15/gP7tFDeJm4yrWc49IwAUTOOiBimggwV LSW3k2u6eZPHUSpH7LgbLnBgMlDQh43XRDgOjOwi9Z1Bjbv4ryVyzzEw8hQ5T1Xi4cUx LK0sJ99oIc8ylQaxPz2GyqFRiyaqkf61koESPLx5hkDgnjlacXD6urKvDZtW02MztEMU VAPw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=aIvG46kE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gb24-20020a170907961800b00a55aead4331si11551966ejc.176.2024.04.30.00.14.17; Tue, 30 Apr 2024 00:14:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b=aIvG46kE; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 9A40E68D5BF; Tue, 30 Apr 2024 10:12:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-221-205.mail.qq.com (out203-205-221-205.mail.qq.com [203.205.221.205]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9D91468D5C2 for ; Tue, 30 Apr 2024 10:12:27 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1714461138; bh=rod3GMoF6TTXU8at+ku8uwfXJdWb2tCpgeC2WwAgris=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=aIvG46kE+sFdjhVYomHGeZa5jpVO+ivdKxL3IYBb5xaRhd1ikzZqqKBEG/ogc8Lvs i+ELst5hI0pklVXxNh8/dZB3yd25G/MQWzFepW5jDSf735xqegVxnJdekBEJSNkRkT 7KB/7Dpd2lyiZh00jxn4kBz/HQpz/rlBV468PSMc= Received: from localhost.localdomain ([113.108.77.51]) by newxmesmtplogicsvrsza10-0.qq.com (NewEsmtp) with SMTP id 3091B0E9; Tue, 30 Apr 2024 15:12:09 +0800 X-QQ-mid: xmsmtpt1714461137t9fwmv0wv Message-ID: X-QQ-XMAILINFO: MQ+wLuVvI2LQq5yd8q1VaGiHqTk+NSulqnKpdEgBpH+3B6MKfdoqXWXimrZgjl Tg9jSqJO1VnJSAukRb6WXOxul0QhkPgASMKOuLlAKM55bc4OtkXlsfxZsgP1Hmlt8/Xxu7iMN28X Hdu6DdiSI5EEs1u97ftxvQDq80SR28J2jV1HUtJR1sRClmcXMQqsazPQg4JXeD1CyRL/iwWXDIow 1dMpRihfCc50pn3LBKWJb+VTfQCA+JlUW3q0asM3bLKFsOCPVFACq/VDQp5kktf/Q5d4EXUJgefd eCOI9ZZMz98zJ/RjpRkDbGwbYhVYu5D+AfwZSPc02K0T+6IPO6gRRch7AwB7CTOnu2ZYEOS11EsS 5i/ors2hExS3zJaoinZh3zkW3wrX3UiANwb1jcBZEPSbyzsFJRhlZnG2CtJrM2DEADPB+6Q7sqs5 Vyzl07/czCRtOoNMZlDbNDzkwpaQUyLqGun9J7uNyGw/o7f6jUkdYUHLSCojAkV8QKfqwLB0AQf1 1G1Ey75Z3Iboyqw4xF90BbF07wgvn0gJ5pOhXKyDjmyTYIbDV3PBqRPSXg+axolupCICVlQK4eUp qEN4s8xmrFMNwUOuX4X53vAyaIRkd3aMhHGq3r3+kqSBDn3IEW8Te4B2+Xzleh7A567kgxCOImsZ UgJdkjlb0EM8V0HRvpYz10c5ZeSRziGotZJJA96VXxz01gWqB+b4zlJHsd0eUpJbEXTibZy+vV62 BWBk/yQveBKrMBNMn86E28zxrK/D2zYZt0zItO0CaeiNzfNZypbuKKa3Zzn3F2a8DOWgfdQJGcjD stD+zSmyySCN9EJJRZijCWzGP9KwfwDEkorEpGWMODk3owgRFjDxz+jiu/oLYQ2hX2tpTgJ2WsKP UCUJ2OzNg4yVqv4RoTlrd/3G5xa+Q5r2YGQH3945tio2TXBwA6uc4NYDZV98IvtGebSFw0YmlSQD MHCyv6Ptx3rlXjw6PqEtVsvvUfEvlkXmyIqIfO/IO0WAZLQe6XgGkkEXMh6MCRHO8fVW5BeeF99N KkJs4rl0VCp2cICW2eQr0tD+3Srek= X-QQ-XMRINFO: MPJ6Tf5t3I/ycC2BItcBVIA= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Tue, 30 Apr 2024 15:12:08 +0800 X-OQ-MSGID: <20240430071208.126817-11-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240430071208.126817-1-quinkblack@foxmail.com> References: <20240430071208.126817-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 10/10] avfilter/vf_dnn_detect: Fix null pointer dereference X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +7DuznpDqYeN From: Zhao Zhili --- libavfilter/vf_dnn_detect.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 926966368a..1830bae181 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -807,11 +807,13 @@ static av_cold void dnn_detect_uninit(AVFilterContext *context) DnnDetectContext *ctx = context->priv; AVDetectionBBox *bbox; ff_dnn_uninit(&ctx->dnnctx); - while(av_fifo_can_read(ctx->bboxes_fifo)) { - av_fifo_read(ctx->bboxes_fifo, &bbox, 1); - av_freep(&bbox); + if (ctx->bboxes_fifo) { + while (av_fifo_can_read(ctx->bboxes_fifo)) { + av_fifo_read(ctx->bboxes_fifo, &bbox, 1); + av_freep(&bbox); + } + av_fifo_freep2(&ctx->bboxes_fifo); } - av_fifo_freep2(&ctx->bboxes_fifo); av_freep(&ctx->anchors); free_detect_labels(ctx); }