From patchwork Fri Apr 30 03:07:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27535 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a11:4023:0:0:0:0 with SMTP id ky35csp2147121pxb; Thu, 29 Apr 2021 20:17:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzs/RQbKrULL27dLiRpmxnzAzmwU//0JfSQXPqtKevmFRJBMtUofVSv4odFODC7Jbjhuway X-Received: by 2002:a17:906:5811:: with SMTP id m17mr1731093ejq.221.1619752644365; Thu, 29 Apr 2021 20:17:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1619752644; cv=none; d=google.com; s=arc-20160816; b=J+kHWQWKL21Fh0N/4v2UykLlTEGsmnwvKgg0gMyPkHLQDTPbU/dPDzXbhVGlRKkRqy JkzqjL9T6ztt22+5ucydWMgIqnv1LardcTjVhoo36H5G1WJeCl2hxnwoBqafP79GbRYm CsonybrJyE+2liybi82D0rRlhgw9NVF+ie7qFnto1Rj0Criq0vX4N/lZiD8DTED5cwli tIB6nKUJJT70SQPGATyBaa7WzN5hOOeuHDZVbvEY7XT736INJriRSPCllmDoU3Vm7ujx 0R04R9yETtpUbV9CD8UBoVZrBHDSVBhBZIKDpSeu9qQ/XxWRrBYpnlvKuaDgtvh2oZYq /UCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=0DndEAXtfDWmziaYdRYMHGe10Txpl4QM4iZB+1WS+gI=; b=LnpkNEcNNmdkEZWWx3akdYR+ksQl1JxF2EaGMMBu0j+RndLksfkB2dSHUnZNkMgfzf XhHCUfsptxzQEUXMVgBhJ8hQUwdHVXJK28gTxUEM9fRwZ/3ygm5kFnV6DH5TOEs0zCot BaR9IyHWEqecpIqJeo0npCEdl+tK+YAyoxybYLt0Ayk+W4BkF9sGsXuj3Mu2YJ7sXJES 3jcbBFXkE5tdkFUNO5KG6E+um2CGgAuFJvWYS1Vww7CqoMFhIIIH7FA2GT9LMYt2Q7MC GMRXyMSedDzuw2Jg0gDMT88jq3JHF2Kqomq3AV3DPt5y0UiIBD90HzENf9WObdRFY0qk 4tdQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hk2si1714381ejb.128.2021.04.29.20.17.24; Thu, 29 Apr 2021 20:17:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 39C8268A2B1; Fri, 30 Apr 2021 06:17:13 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9D6426807AA for ; Fri, 30 Apr 2021 06:17:06 +0300 (EEST) IronPort-SDR: QTSex/YD3TwB0SgVhdO/SuC1k8AMnSIzH9ppQPLAl7xN7u8O60uhlbFXCGPMonAST2/cj9BIPr oW4UP4C5nCxg== X-IronPort-AV: E=McAfee;i="6200,9189,9969"; a="184659710" X-IronPort-AV: E=Sophos;i="5.82,260,1613462400"; d="scan'208";a="184659710" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Apr 2021 20:16:57 -0700 IronPort-SDR: G7tctzc07GXeLnq65KTx0fY6W+FtK1hr6/mxdgk3lSnYlBc6aoplQ6kGPTdcLfFczniJfEelh4 WdhH+Y7K80VA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,260,1613462400"; d="scan'208";a="459152502" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by fmsmga002.fm.intel.com with ESMTP; 29 Apr 2021 20:16:56 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 30 Apr 2021 11:07:09 +0800 Message-Id: <20210430030711.30216-2-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210430030711.30216-1-ting.fu@intel.com> References: <20210430030711.30216-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 2/4] lavfi/dnn_backend_tensorflow: add multiple outputs support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Heb170h0Wszq Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_backend_tf.c | 49 ++++++++++++++++---------------- libavfilter/dnn_filter_common.c | 45 +++++++++++++++++++++++++---- libavfilter/dnn_filter_common.h | 6 ++-- libavfilter/vf_derain.c | 2 +- libavfilter/vf_sr.c | 2 +- 5 files changed, 71 insertions(+), 33 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index f70e3d4659..5c85b562c4 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -155,7 +155,7 @@ static DNNReturnType get_input_tf(void *model, DNNData *input, const char *input TF_DeleteStatus(status); // currently only NHWC is supported - av_assert0(dims[0] == 1); + av_assert0(dims[0] == 1 || dims[0] == -1); input->height = dims[1]; input->width = dims[2]; input->channels = dims[3]; @@ -707,7 +707,7 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n TF_Output *tf_outputs; TFModel *tf_model = model->model; TFContext *ctx = &tf_model->ctx; - DNNData input, output; + DNNData input, *outputs; TF_Tensor **output_tensors; TF_Output tf_input; TF_Tensor *input_tensor; @@ -738,14 +738,6 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n } } - if (nb_output != 1) { - // currently, the filter does not need multiple outputs, - // so we just pending the support until we really need it. - TF_DeleteTensor(input_tensor); - avpriv_report_missing_feature(ctx, "multiple outputs"); - return DNN_ERROR; - } - tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs)); if (tf_outputs == NULL) { TF_DeleteTensor(input_tensor); @@ -785,23 +777,31 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n return DNN_ERROR; } + outputs = av_malloc_array(nb_output, sizeof(*outputs)); + if (!outputs) { + TF_DeleteTensor(input_tensor); + av_freep(&tf_outputs); + av_freep(&output_tensors); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); \ + return DNN_ERROR; + } + for (uint32_t i = 0; i < nb_output; ++i) { - output.height = TF_Dim(output_tensors[i], 1); - output.width = TF_Dim(output_tensors[i], 2); - output.channels = TF_Dim(output_tensors[i], 3); - output.data = TF_TensorData(output_tensors[i]); - output.dt = TF_TensorType(output_tensors[i]); - - if (do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(out_frame, &output, tf_model->model->filter_ctx); - } else { - ff_proc_from_dnn_to_frame(out_frame, &output, ctx); - } + outputs[i].height = TF_Dim(output_tensors[i], 1); + outputs[i].width = TF_Dim(output_tensors[i], 2); + outputs[i].channels = TF_Dim(output_tensors[i], 3); + outputs[i].data = TF_TensorData(output_tensors[i]); + outputs[i].dt = TF_TensorType(output_tensors[i]); + } + if (do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(out_frame, outputs, tf_model->model->filter_ctx); } else { - out_frame->width = output.width; - out_frame->height = output.height; + ff_proc_from_dnn_to_frame(out_frame, outputs, ctx); } + } else { + out_frame->width = outputs[0].width; + out_frame->height = outputs[0].height; } for (uint32_t i = 0; i < nb_output; ++i) { @@ -812,6 +812,7 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n TF_DeleteTensor(input_tensor); av_freep(&output_tensors); av_freep(&tf_outputs); + av_freep(&outputs); return DNN_SUCCESS; } diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index 1b922455a3..4cbfdbf52a 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -17,6 +17,39 @@ */ #include "dnn_filter_common.h" +#include "libavutil/avstring.h" + +#define MAX_SUPPORTED_OUTPUTS_NB 4 + +static char **separate_output_names(const char *expr, const char *val_sep, int *separated_nb) +{ + char *val, **parsed_vals = NULL; + int val_num = 0; + if (!expr || !val_sep || !separated_nb) { + return NULL; + } + + parsed_vals = av_mallocz_array(MAX_SUPPORTED_OUTPUTS_NB, sizeof(*parsed_vals)); + if (!parsed_vals) { + return NULL; + } + + do { + val = av_get_token(&expr, val_sep); + if(val) { + parsed_vals[val_num] = val; + val_num++; + } + if (*expr) { + expr++; + } + } while(*expr); + + parsed_vals[val_num] = NULL; + *separated_nb = val_num; + + return parsed_vals; +} int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx) { @@ -28,8 +61,10 @@ int ff_dnn_init(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *fil av_log(filter_ctx, AV_LOG_ERROR, "input name of the model network is not specified\n"); return AVERROR(EINVAL); } - if (!ctx->model_outputname) { - av_log(filter_ctx, AV_LOG_ERROR, "output name of the model network is not specified\n"); + + ctx->model_outputnames = separate_output_names(ctx->model_outputnames_string, "&", &ctx->nb_outputs); + if (!ctx->model_outputnames) { + av_log(filter_ctx, AV_LOG_ERROR, "could not parse model output names\n"); return AVERROR(EINVAL); } @@ -85,19 +120,19 @@ DNNReturnType ff_dnn_get_input(DnnContext *ctx, DNNData *input) DNNReturnType ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height) { return ctx->model->get_output(ctx->model->model, ctx->model_inputname, input_width, input_height, - ctx->model_outputname, output_width, output_height); + (const char *)ctx->model_outputnames[0], output_width, output_height); } DNNReturnType ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame) { return (ctx->dnn_module->execute_model)(ctx->model, ctx->model_inputname, in_frame, - (const char **)&ctx->model_outputname, 1, out_frame); + (const char **)ctx->model_outputnames, ctx->nb_outputs, out_frame); } DNNReturnType ff_dnn_execute_model_async(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame) { return (ctx->dnn_module->execute_model_async)(ctx->model, ctx->model_inputname, in_frame, - (const char **)&ctx->model_outputname, 1, out_frame); + (const char **)ctx->model_outputnames, ctx->nb_outputs, out_frame); } DNNAsyncStatusType ff_dnn_get_async_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame) diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h index 8deb18b39a..586d2afed4 100644 --- a/libavfilter/dnn_filter_common.h +++ b/libavfilter/dnn_filter_common.h @@ -30,10 +30,12 @@ typedef struct DnnContext { char *model_filename; DNNBackendType backend_type; char *model_inputname; - char *model_outputname; + char *model_outputnames_string; + uint32_t nb_outputs; char *backend_options; int async; + char **model_outputnames; DNNModule *dnn_module; DNNModel *model; } DnnContext; @@ -41,7 +43,7 @@ typedef struct DnnContext { #define DNN_COMMON_OPTIONS \ { "model", "path to model file", OFFSET(model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ { "input", "input name of the model", OFFSET(model_inputname), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ - { "output", "output name of the model", OFFSET(model_outputname), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ + { "output", "output name of the model", OFFSET(model_outputnames_string), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ { "backend_configs", "backend configs", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ { "options", "backend configs", OFFSET(backend_options), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },\ { "async", "use DNN async inference", OFFSET(async), AV_OPT_TYPE_BOOL, { .i64 = 1}, 0, 1, FLAGS}, diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index 76c4ef414f..5037f3a5f7 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -50,7 +50,7 @@ static const AVOption derain_options[] = { #endif { "model", "path to model file", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputname), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, + { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } }; diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 4360439ca6..f930b38748 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -54,7 +54,7 @@ static const AVOption sr_options[] = { { "scale_factor", "scale factor for SRCNN model", OFFSET(scale_factor), AV_OPT_TYPE_INT, { .i64 = 2 }, 2, 4, FLAGS }, { "model", "path to model file specifying network architecture and its parameters", OFFSET(dnnctx.model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, { "input", "input name of the model", OFFSET(dnnctx.model_inputname), AV_OPT_TYPE_STRING, { .str = "x" }, 0, 0, FLAGS }, - { "output", "output name of the model", OFFSET(dnnctx.model_outputname), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, + { "output", "output name of the model", OFFSET(dnnctx.model_outputnames_string), AV_OPT_TYPE_STRING, { .str = "y" }, 0, 0, FLAGS }, { NULL } };