From patchwork Mon Jul 5 10:30:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28781 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4025953ios; Mon, 5 Jul 2021 03:31:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwPGjNhyAkr1Yqpm0haevwABWgRqbqZRNMUku7VIGlE9ADoCgoFJLIYd/Y4JDMJX0jPnGux X-Received: by 2002:a17:906:2642:: with SMTP id i2mr12897323ejc.323.1625481101310; Mon, 05 Jul 2021 03:31:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481101; cv=none; d=google.com; s=arc-20160816; b=xGKihlCrsw7PG7DVpCFZ5I5SdHctg79cpHmovFMDvFMr7PVTtz4SQ3GVVGEOxWSCKe mrcNl1DpcQEg55vo811GQ1exK006jNPEdJPwRBJPqZPxg+L7qWgWq9wnz7Z2VRXKi4Wn nn6vPHxjUT406b9ixb6LuqseghkNDL8bLSh+nmI6cWJjfWdWx4wa09jnp4qx1f2I9e2b nKHqW6xrFhKpRSog1mTc+x4TCpE4xAAE/i/xawW5Bnfvtpz8MVnP47PT6CwDQtlJdsCL /PvtDrIh3xir3mXIhOqz3ttaowkbXdDDg6oteCG5G4L3RMm86uTT45qXXFt8uA3ODHJp quVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ok4QFZZnmYWzK66aMi5bdWz3CqG57rRa8SFeDqOY+Og=; b=TNNx0uCyhsqi6jBr/OG0vqCnR7Jzod2OclySfvolUmyHEeMj1MTPiPhPGvxBQ8QqY1 xSU3bYbIbbkZ0zc0mNCU5I6cqASczRTW7NzSePrIwYfi7uUCOschBKN8bQ0qMId+624N FAuuNmd5t0YcI7DiJJtAAyC35eofxDqLd4LloddR9ZQ+pVHAMSggkVIvqqrvPJbaOfzC oRrT0GsTQiAEMAiKKpTgrkLLGe3MTmRFvt7qHsrr0PYTxeTvIlX2hv3yJISntqe74W2l PewiMPbfdXq7LaBmSW5GhzYNkZRsQYtCGhC4HFd2YOJLeaiPUGPlcAT3EVWM6+ds5fMF uMYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=do6KVfzW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gt41si1875263ejc.192.2021.07.05.03.31.39; Mon, 05 Jul 2021 03:31:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=do6KVfzW; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2FC17689EB2; Mon, 5 Jul 2021 13:31:36 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A4CD068A307 for ; Mon, 5 Jul 2021 13:31:29 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id i13so9975841plb.10 for ; Mon, 05 Jul 2021 03:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=X1OPuerU6E1VMcYSpgrWG9hNQqnWVP/1z0aZxqhuadM=; b=do6KVfzWQ7ukC+QWQ21ODNTSQESCZmjoFyB/9lJIs/Zn68cTT0fixlVuXyn7k1jXwt IaNG/jOIv1Wo0ILA5ZWmnVGO9y7iL59GdhLN+m8kR4sHFW4VZCV32R/YudYgUV+LOmKS 99qan3PD+XcZz9uj/P6UddQDJUXxBbN+HaUlsjDqP9ugUgicUzTtjZvAW/3I4W71lAXC LPS/nnD3YQc1iNFWmh7qlNNp9bIH5npbgso39tTUSf/epiv68V3fyu9cu6yvvKMxNUIK Yr9sYj/Ir8I/nusExq9+OfhrOEG3mgeVHHZ7CTHIqsmvR5gcNAUBiJ9xnI9bDsh4TDHu eVmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=X1OPuerU6E1VMcYSpgrWG9hNQqnWVP/1z0aZxqhuadM=; b=b7EKblqOBDOGyP4nr9IXaoBlszxBN2DGELw0xLlwKHdLlWeqpsGrqiP9gr4K/h8uHv 5sO9a99lrb/qGCl3R2XNc/2o6ClkzecyhczWGTk6D8YegD+l9a9CYcuK/DnvT8mu6myU VhAHb3GhCa8E1knVaIo1b22LoV9hSaGnPhd2RinWMuSNKlH33dX8mcTahVyJUqz5GtXX oWmUlS18cTT8qfZdSFSKmzLo/DW+artrsK9X9TexhpwIC4n7+NSFzKuHtXhoCWowlip8 5jS+kuxABBwODUsnKHP8GY0vGz8uP0Q55+j7/EyFMmelpaD/vCW/vYgYmxJxebP1qhUL J+Ig== X-Gm-Message-State: AOAM5303VHTZ3O6drYxA2qNbNAx1HUmrbeoz+eL0UHcrdGgijsbZi1+K VR+wJgi7dPumgk0/naXPQRFNfK/i4uEKow== X-Received: by 2002:a17:90a:af90:: with SMTP id w16mr14890406pjq.129.1625481087187; Mon, 05 Jul 2021 03:31:27 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:26 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:53 +0530 Message-Id: <20210705103057.42309-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 1/6] lavfi/dnn_backend_tf: TaskItem Based Inference X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: h03ODZfer9sU This commit uses the common TaskItem and InferenceItem typedefs for execution in TensorFlow backend. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 134 ++++++++++++++++++++++--------- 1 file changed, 94 insertions(+), 40 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 4c16c2bdb0..8762211ebc 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -35,6 +35,7 @@ #include "dnn_backend_native_layer_maximum.h" #include "dnn_io_proc.h" #include "dnn_backend_common.h" +#include "queue.h" #include typedef struct TFOptions{ @@ -52,6 +53,7 @@ typedef struct TFModel{ TF_Graph *graph; TF_Session *session; TF_Status *status; + Queue *inference_queue; } TFModel; #define OFFSET(x) offsetof(TFContext, x) @@ -63,15 +65,29 @@ static const AVOption dnn_tensorflow_options[] = { AVFILTER_DEFINE_CLASS(dnn_tensorflow); -static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_name, AVFrame *in_frame, - const char **output_names, uint32_t nb_output, AVFrame *out_frame, - int do_ioproc); +static DNNReturnType execute_model_tf(Queue *inference_queue); static void free_buffer(void *data, size_t length) { av_freep(&data); } +static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) +{ + InferenceItem *inference = av_malloc(sizeof(*inference)); + if (!inference) { + return DNN_ERROR; + } + task->inference_todo = 1; + task->inference_done = 0; + inference->task = task; + if (ff_queue_push_back(inference_queue, inference) < 0) { + av_freep(&inference); + return DNN_ERROR; + } + return DNN_SUCCESS; +} + static TF_Buffer *read_graph(const char *model_filename) { TF_Buffer *graph_buf; @@ -171,6 +187,7 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu TFContext *ctx = &tf_model->ctx; AVFrame *in_frame = av_frame_alloc(); AVFrame *out_frame = NULL; + TaskItem task; if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); @@ -187,7 +204,21 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu in_frame->width = input_width; in_frame->height = input_height; - ret = execute_model_tf(tf_model->model, input_name, in_frame, &output_name, 1, out_frame, 0); + task.do_ioproc = 0; + task.async = 0; + task.input_name = input_name; + task.in_frame = in_frame; + task.output_names = &output_name; + task.out_frame = out_frame; + task.model = tf_model; + task.nb_output = 1; + + if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + ret = execute_model_tf(tf_model->inference_queue); *output_width = out_frame->width; *output_height = out_frame->height; @@ -723,6 +754,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } } + tf_model->inference_queue = ff_queue_create(); model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -733,26 +765,33 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_name, AVFrame *in_frame, - const char **output_names, uint32_t nb_output, AVFrame *out_frame, - int do_ioproc) +static DNNReturnType execute_model_tf(Queue *inference_queue) { TF_Output *tf_outputs; - TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + TFModel *tf_model; + TFContext *ctx; + InferenceItem *inference; + TaskItem *task; DNNData input, *outputs; TF_Tensor **output_tensors; TF_Output tf_input; TF_Tensor *input_tensor; - if (get_input_tf(tf_model, &input, input_name) != DNN_SUCCESS) + inference = ff_queue_pop_front(inference_queue); + av_assert0(inference); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + + if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) return DNN_ERROR; - input.height = in_frame->height; - input.width = in_frame->width; - tf_input.oper = TF_GraphOperationByName(tf_model->graph, input_name); + input.height = task->in_frame->height; + input.width = task->in_frame->width; + + tf_input.oper = TF_GraphOperationByName(tf_model->graph, task->input_name); if (!tf_input.oper){ - av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name); + av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); return DNN_ERROR; } tf_input.index = 0; @@ -765,30 +804,30 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: - if (do_ioproc) { + if (task->do_ioproc) { if (tf_model->model->frame_pre_proc != NULL) { - tf_model->model->frame_pre_proc(in_frame, &input, tf_model->model->filter_ctx); + tf_model->model->frame_pre_proc(task->in_frame, &input, tf_model->model->filter_ctx); } else { - ff_proc_from_frame_to_dnn(in_frame, &input, ctx); + ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } } break; case DFT_ANALYTICS_DETECT: - ff_frame_to_dnn_detect(in_frame, &input, ctx); + ff_frame_to_dnn_detect(task->in_frame, &input, ctx); break; default: avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model->func_type); break; } - tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs)); + tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); if (tf_outputs == NULL) { TF_DeleteTensor(input_tensor); av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); \ return DNN_ERROR; } - output_tensors = av_mallocz_array(nb_output, sizeof(*output_tensors)); + output_tensors = av_mallocz_array(task->nb_output, sizeof(*output_tensors)); if (!output_tensors) { TF_DeleteTensor(input_tensor); av_freep(&tf_outputs); @@ -796,13 +835,13 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n return DNN_ERROR; } - for (int i = 0; i < nb_output; ++i) { - tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, output_names[i]); + for (int i = 0; i < task->nb_output; ++i) { + tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); if (!tf_outputs[i].oper) { TF_DeleteTensor(input_tensor); av_freep(&tf_outputs); av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", output_names[i]); \ + av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); \ return DNN_ERROR; } tf_outputs[i].index = 0; @@ -810,7 +849,7 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n TF_SessionRun(tf_model->session, NULL, &tf_input, &input_tensor, 1, - tf_outputs, output_tensors, nb_output, + tf_outputs, output_tensors, task->nb_output, NULL, 0, NULL, tf_model->status); if (TF_GetCode(tf_model->status) != TF_OK) { TF_DeleteTensor(input_tensor); @@ -820,7 +859,7 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n return DNN_ERROR; } - outputs = av_malloc_array(nb_output, sizeof(*outputs)); + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { TF_DeleteTensor(input_tensor); av_freep(&tf_outputs); @@ -829,36 +868,36 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n return DNN_ERROR; } - for (uint32_t i = 0; i < nb_output; ++i) { + for (uint32_t i = 0; i < task->nb_output; ++i) { outputs[i].height = TF_Dim(output_tensors[i], 1); outputs[i].width = TF_Dim(output_tensors[i], 2); outputs[i].channels = TF_Dim(output_tensors[i], 3); outputs[i].data = TF_TensorData(output_tensors[i]); outputs[i].dt = TF_TensorType(output_tensors[i]); } - switch (model->func_type) { + switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: //it only support 1 output if it's frame in & frame out - if (do_ioproc) { + if (task->do_ioproc) { if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(out_frame, outputs, tf_model->model->filter_ctx); + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); } else { - ff_proc_from_dnn_to_frame(out_frame, outputs, ctx); + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } } else { - out_frame->width = outputs[0].width; - out_frame->height = outputs[0].height; + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; } break; case DFT_ANALYTICS_DETECT: - if (!model->detect_post_proc) { + if (!tf_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return DNN_ERROR; } - model->detect_post_proc(out_frame, outputs, nb_output, model->filter_ctx); + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; default: - for (uint32_t i = 0; i < nb_output; ++i) { + for (uint32_t i = 0; i < task->nb_output; ++i) { if (output_tensors[i]) { TF_DeleteTensor(output_tensors[i]); } @@ -871,30 +910,39 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); return DNN_ERROR; } - - for (uint32_t i = 0; i < nb_output; ++i) { + for (uint32_t i = 0; i < task->nb_output; ++i) { if (output_tensors[i]) { TF_DeleteTensor(output_tensors[i]); } } + task->inference_done++; TF_DeleteTensor(input_tensor); av_freep(&output_tensors); av_freep(&tf_outputs); av_freep(&outputs); return DNN_SUCCESS; + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { TFModel *tf_model = model->model; TFContext *ctx = &tf_model->ctx; + TaskItem task; if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { - return DNN_ERROR; + return DNN_ERROR; } - return execute_model_tf(model, exec_params->input_name, exec_params->in_frame, - exec_params->output_names, exec_params->nb_output, exec_params->out_frame, 1); + if (ff_dnn_fill_task(&task, exec_params, tf_model, 0, 1) != DNN_SUCCESS) { + return DNN_ERROR; + } + + if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + return execute_model_tf(tf_model->inference_queue); } void ff_dnn_free_model_tf(DNNModel **model) @@ -903,6 +951,12 @@ void ff_dnn_free_model_tf(DNNModel **model) if (*model){ tf_model = (*model)->model; + while (ff_queue_size(tf_model->inference_queue) != 0) { + InferenceItem *item = ff_queue_pop_front(tf_model->inference_queue); + av_freep(&item); + } + ff_queue_destroy(tf_model->inference_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } From patchwork Mon Jul 5 10:30:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28784 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4031721ios; Mon, 5 Jul 2021 03:39:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwL1bnRRCgTjJJpXvmd7t5TEiC9qoidFWIIYL6YdCrGO+QoFPihBdpqyUzKMf8Zf5eCvrTU X-Received: by 2002:a05:6402:88b:: with SMTP id e11mr15370287edy.21.1625481589689; Mon, 05 Jul 2021 03:39:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481589; cv=none; d=google.com; s=arc-20160816; b=Mw7LOCVS5x60x7+u72SpXKtktI1LtnvdVLVl9+F1o7PjB4TtrNJ53SBo4HbErLhK9d RrMz09t7y6fbQlz/MSJE/22Axu1W6Zsc6lwm2Rza1O2SeT3dOy2OhXBNHsxD6iG+n6Zq VZIatZdVFHac9JMznokcV1dbugpqEmOM27B9fbhycVoa7Za1Es6FSYoHMWa89NsVWnTm 4iHkjsCuQjCAkppGG6IFb7RlbcB9sH3MLjLybU0nIcjBzZ6zn6dazMXTnEnQoX/+zZi4 uzEEdXTEIky00PqIR00eH5Sjx7Lgz7KRFxnyEWUoiOE4iarvCpIy3l4TNN7L117qaZpN Qn6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=RpB6P6ODI9OWTQu78a/MBC8TvuDjm15fBCP1ESXLPak=; b=hwSjg1QWn2PGX8ZVQ+PuKLWH6yGLxoJP9PNMahMT/KPAIU3SqvTlUkThsN0ofWCxg2 Vro/XdgDod4sICjNZlFpW6O1xY+oO4Ezk5uHibI75l1D0f108bRYeoMS0WANITyfMCQa BJieYp5ZDNUuiBgMjsSU3iFYWHuDqJKRAhrS5epD0MRud6qmTJoaP9hAjGYtKrGXENuW MTslwU8C9CcCv9fgRg6dxGMi+MFEZpN9whYipxkMdBSwBBo/j1CLfQzmihPYgiGQsJAr fJ8J2/UpK4dTb36by95ruxYDSgm5kWzuuJvkSTZFPNTc/UnXrcFh8hCFBtJc2jySlxCQ lbMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=uAiZiric; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hs13si4639665ejc.150.2021.07.05.03.39.49; Mon, 05 Jul 2021 03:39:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=uAiZiric; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 82C1068A6B5; Mon, 5 Jul 2021 13:39:47 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4897D68A5D7 for ; Mon, 5 Jul 2021 13:39:41 +0300 (EEST) Received: by mail-pf1-f179.google.com with SMTP id x16so16304574pfa.13 for ; Mon, 05 Jul 2021 03:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y73bDz8K09pPg7xhjEErY122Ikc8CnH1/Z6WXm/Ad68=; b=uAiZiric4FlaTLdgAKP1J3bHFDgq4uaXuJnN7HQaIYpXoyoxg97o8oU3xeZN9P5yEk PtfnS+5sVkzv4MF6Z4Y9UPhplew8lSgWOZIRy12o0psQQfg179xAbzXKWeUBWlcBejDU lz12X1csLKvZsx4vyOMdnQ/pBbUj9fdZQFAw5kELmeyMR/o+DI2UEpAKqM1j5BYrOV/O 1CaknmQETrKY7jrKmbG8I2kNtv8/4y+Ude+KtK59iXVAzyG15fbJA9GVSqmlRxqfXSDK XbA0FjS9g9dxJKpId8EcpcBXR7ykQw+BrX3xM7zHWkPh1O6IShJcMwB3ahbxoW+3RrnA fc2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y73bDz8K09pPg7xhjEErY122Ikc8CnH1/Z6WXm/Ad68=; b=Rm3Pzh9dv/W1THSV5iiPMZ4CWgqj0+svcSpyOvlTo6ol7ecN7nKangcuJqTDZrn3sI lKk0y2VYHAoVGHTw/pUMjWUuQvAN+C9a+FV5NSi8XtLecCooELEsfIZALh4fGf1B+84e RWlLdnwLVhMdT4VkZ5HuTTWirDVvoKkn62gkHpHCqh5ni1G6GyXNPRpAKgtc96kXZDKk WtOfLa3YwDTP1ChFep5IxhqKY+QQBqyoiSUb14u+Ax/TFTyIhLXzcom7XkygIzF1LWGK /GI6hihmFMkVnc6cAi2C76g8xWFQ06wQGR/kaIw42f1saICfkU9k2vlyBTu6bM2/qqwf zZqQ== X-Gm-Message-State: AOAM530mUqa0DsDP5EPsKmYt+464+GMNjpu56v8cPOSX+VCbgpyYFSya ZpH2CMjWETrZvc4+N7201b2VAKZ00QIcPw== X-Received: by 2002:a17:903:2452:b029:129:33a0:399d with SMTP id l18-20020a1709032452b029012933a0399dmr12163988pls.31.1625481089095; Mon, 05 Jul 2021 03:31:29 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:28 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:54 +0530 Message-Id: <20210705103057.42309-2-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 2/6] lavfi/dnn_backend_tf: Add TFInferRequest and TFRequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: m9ystMfelAFj This commit introduces a typedef TFInferRequest to store execution parameters for a single call to the TensorFlow C API. This typedef is used in the TFRequestItem. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 49 ++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 8762211ebc..578748eb35 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -56,6 +56,23 @@ typedef struct TFModel{ Queue *inference_queue; } TFModel; +/** + * Stores execution parameters for single + * call to the TensorFlow C API + */ +typedef struct TFInferRequest { + TF_Output *tf_outputs; + TF_Tensor **output_tensors; + TF_Output *tf_input; + TF_Tensor *input_tensor; +} TFInferRequest; + +typedef struct TFRequestItem { + TFInferRequest *infer_request; + InferenceItem *inference; + // further properties will be added later for async +} TFRequestItem; + #define OFFSET(x) offsetof(TFContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_tensorflow_options[] = { @@ -72,6 +89,38 @@ static void free_buffer(void *data, size_t length) av_freep(&data); } +static void tf_free_request(TFInferRequest *request) +{ + if (!request) + return; + if (request->input_tensor) { + TF_DeleteTensor(request->input_tensor); + request->input_tensor = NULL; + } + av_freep(&request->tf_input); + av_freep(&request->tf_outputs); + if (request->output_tensors) { + int nb_output = sizeof(*request->output_tensors)/sizeof(request->output_tensors[0]); + for (uint32_t i = 0; i < nb_output; ++i) { + if (request->output_tensors[i]) { + TF_DeleteTensor(request->output_tensors[i]); + request->output_tensors[i] = NULL; + } + } + av_freep(&request->output_tensors); + } +} + +static TFInferRequest *tf_create_inference_request(void) +{ + TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); + infer_request->tf_outputs = NULL; + infer_request->tf_input = NULL; + infer_request->input_tensor = NULL; + infer_request->output_tensors = NULL; + return infer_request; +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { InferenceItem *inference = av_malloc(sizeof(*inference)); From patchwork Mon Jul 5 10:30:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28785 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4042383ios; Mon, 5 Jul 2021 03:56:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw6SP0Og0yATGsZS1tfo5e8EYuDC8Lh4yu/mLsAHctfZ5IgbCVS9wmOr52cseDYzrgDr/Xf X-Received: by 2002:a17:907:2ce1:: with SMTP id hz1mr12473538ejc.376.1625482581987; Mon, 05 Jul 2021 03:56:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625482581; cv=none; d=google.com; s=arc-20160816; b=UyImzY7XF+Tu3TWq9zsC35jrWreRF4DUxeLDeUeJVADZx+jS4iO9c1IDv7W3lOx0Hw YoEYpDV66Oc5RsOmwpOiN+TGDgNLOQrsYgTK/yKtftAzaVkeSJ8ZRtpZ/CudRBP6Rfvt XM6MXPTI5Rm4DonkhmJojHiOvNET3zCjNUiL1mWSkoaXg+TSfJDJ4puAP+5juD5iKolc q3aurNw2A9bjsTeLlVXtbnJ2hkLsdkhSOQISNEB4Mje7NTskqxmVeIEFiIxV0kPehC8U ki4Aw6R6mrXdQUGjM8UTH1WJR6tIxs4X4Vo28MO5H2aCgJVo8wV8XBbDXIL3TlRtUTD6 zkVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=re+dprN96msf+Gdm9Xs7Rk1DPh2ULmJUlEuYnCaj3+w=; b=y7M59O0p8XocOvDiwSzPw1sHnx0IuDiRoz86m5hrJGO7KSmyVXYrTpNBdnI07p2wOA 9veUIzb66w/fgRCtiFKdjbjHH7Ukz52o+h4BZyWQ6KB/5GHxTwdvwGA7GJlWHDIklxmA G/qmN4Z1FsSKW/ds2j55PlY8EfVObTu5KrhIEp1w9rMuOwBncaS2BjysVN0vDewCd935 ivlXjIbKwncDOTDsdpXNWlLU0iL2ng5t+1wlHbjLThzwLwIE6Fsiawfcwr7boNunx2fV hB90hNCSV70/xc5mOrwK9nVGqrIrHxw1wOAtBSUioKa8vW7mp8UfGsEVS5AMgGtGlGgX 6rOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=G1S4c578; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id co5si10812928edb.500.2021.07.05.03.56.21; Mon, 05 Jul 2021 03:56:21 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=G1S4c578; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EB71A68A66B; Mon, 5 Jul 2021 13:56:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 64AC068A334 for ; Mon, 5 Jul 2021 13:56:11 +0300 (EEST) Received: by mail-pf1-f172.google.com with SMTP id w22so12721930pff.5 for ; Mon, 05 Jul 2021 03:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CsuiDPyHVptShydgmMhhYcg4B0oOyOEf2isfwjRBjOc=; b=G1S4c578MO9HZnnqgQHlAhrWx8y5PmVpD8NQ4t4qAzM5ZspNWcHvGSjU7uLJQKBc7y SKyLCPrCN91TiV7nL7xD+hHZHTfPHPDaayIaMDi1igjz1aMBakxywKo81mXoaemXfzKN R1SqfCRcD/VQVKpuDjDzGcC/JQBUm3D8VLjL8wHd4ECQnMA4mxUs2M7DifSA4M/39a9a YSnnV95uBrfwLmWPAXfg8l7VGPz1oyS7kyPjp7dCB1LquT8RVOGWifgXpMPTSuSU24fc dEmn7NrH6wnE0lgLOjVX9UPKjQ2d+hVznZwptdAb1sRzsYay7vqFeW1f5wpQbEXchoMU 3Gag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CsuiDPyHVptShydgmMhhYcg4B0oOyOEf2isfwjRBjOc=; b=AdNi5apXu5KhGAC7Vml0EsacSLXIQY2AhG8+uu6FfFTNn5BRqkd8Pra2mAL1qgM9F9 ilws7LfCna7vp0obVIHJdwhS3s3WspnMH2a0V3WJaU9DX3aY3J9Y/eFRvb71iG/bLeVQ Za6hY+bpI31ti1rsjUZ/dWmS3C/t2EDsZGjkdUQDJnv9AxT2yRArSiq8G/iMAuzgDAP7 SFAOzeJmxIkjCj9ZxtJjg+uiRDsvzp+v+kP8pQiaXVgZ9lJZjUQ7TNwgM+mQGaCtljmZ cEH0A3kmnz5R1XA2rHmnv7WhQ/I8HZMVWA+8jwGBQ7P1biEei6DDMSVR48fSrzq0qExG x/lw== X-Gm-Message-State: AOAM532qCpu6Jcy9+6L8GvP/7IrP059DUFWtYNdgnNzxkQNVDyycGc2u ZA+tXyCdLlmKJt5/olnYSUC7oHOkHhMO0A== X-Received: by 2002:a63:383:: with SMTP id 125mr9799643pgd.208.1625481091429; Mon, 05 Jul 2021 03:31:31 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:31 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:55 +0530 Message-Id: <20210705103057.42309-3-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 3/6] lavfi/dnn_backend_tf: Request-based Execution X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: N5BRkDqQF/Jy This commit uses TFRequestItem and the existing sync execution mechanism to use request-based execution. It will help in adding async functionality to the TensorFlow backend later. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 3 + libavfilter/dnn/dnn_backend_openvino.c | 2 +- libavfilter/dnn/dnn_backend_tf.c | 156 ++++++++++++++----------- 3 files changed, 91 insertions(+), 70 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index df59615f40..5281fdfed1 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -26,6 +26,9 @@ #include "../dnn_interface.h" +#define DNN_BACKEND_COMMON_OPTIONS \ + { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + // one task for one function call from dnn interface typedef struct TaskItem { void *model; // model for the backend diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 3295fc79d3..f34b8150f5 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -75,7 +75,7 @@ typedef struct RequestItem { #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_openvino_options[] = { { "device", "device to run model", OFFSET(options.device_type), AV_OPT_TYPE_STRING, { .str = "CPU" }, 0, 0, FLAGS }, - { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, + DNN_BACKEND_COMMON_OPTIONS { "batch_size", "batch size per request", OFFSET(options.batch_size), AV_OPT_TYPE_INT, { .i64 = 1 }, 1, 1000, FLAGS}, { "input_resizable", "can input be resizable or not", OFFSET(options.input_resizable), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, { NULL } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 578748eb35..e8007406c8 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -35,11 +35,13 @@ #include "dnn_backend_native_layer_maximum.h" #include "dnn_io_proc.h" #include "dnn_backend_common.h" +#include "safe_queue.h" #include "queue.h" #include typedef struct TFOptions{ char *sess_config; + uint32_t nireq; } TFOptions; typedef struct TFContext { @@ -53,6 +55,7 @@ typedef struct TFModel{ TF_Graph *graph; TF_Session *session; TF_Status *status; + SafeQueue *request_queue; Queue *inference_queue; } TFModel; @@ -77,12 +80,13 @@ typedef struct TFRequestItem { #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_tensorflow_options[] = { { "sess_config", "config for SessionOptions", OFFSET(options.sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, + DNN_BACKEND_COMMON_OPTIONS { NULL } }; AVFILTER_DEFINE_CLASS(dnn_tensorflow); -static DNNReturnType execute_model_tf(Queue *inference_queue); +static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue); static void free_buffer(void *data, size_t length) { @@ -237,6 +241,7 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu AVFrame *in_frame = av_frame_alloc(); AVFrame *out_frame = NULL; TaskItem task; + TFRequestItem *request; if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); @@ -267,7 +272,13 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu return DNN_ERROR; } - ret = execute_model_tf(tf_model->inference_queue); + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = execute_model_tf(request, tf_model->inference_queue); *output_width = out_frame->width; *output_height = out_frame->height; @@ -771,6 +782,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ { DNNModel *model = NULL; TFModel *tf_model = NULL; + TFContext *ctx = NULL; model = av_mallocz(sizeof(DNNModel)); if (!model){ @@ -782,13 +794,14 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&model); return NULL; } - tf_model->ctx.class = &dnn_tensorflow_class; tf_model->model = model; + ctx = &tf_model->ctx; + ctx->class = &dnn_tensorflow_class; //parse options - av_opt_set_defaults(&tf_model->ctx); - if (av_opt_set_from_string(&tf_model->ctx, options, NULL, "=", "&") < 0) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); + av_opt_set_defaults(ctx); + if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { + av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); av_freep(&tf_model); av_freep(&model); return NULL; @@ -803,6 +816,18 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } } + if (ctx->options.nireq <= 0) { + ctx->options.nireq = av_cpu_count() / 2 + 1; + } + + tf_model->request_queue = ff_safe_queue_create(); + + for (int i = 0; i < ctx->options.nireq; i++) { + TFRequestItem *item = av_mallocz(sizeof(*item)); + item->infer_request = tf_create_inference_request(); + ff_safe_queue_push_back(tf_model->request_queue, item); + } + tf_model->inference_queue = ff_queue_create(); model->model = tf_model; model->get_input = &get_input_tf; @@ -814,42 +839,42 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(Queue *inference_queue) +static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) { - TF_Output *tf_outputs; TFModel *tf_model; TFContext *ctx; + TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; DNNData input, *outputs; - TF_Tensor **output_tensors; - TF_Output tf_input; - TF_Tensor *input_tensor; inference = ff_queue_pop_front(inference_queue); av_assert0(inference); task = inference->task; tf_model = task->model; ctx = &tf_model->ctx; + request->inference = inference; if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) return DNN_ERROR; + infer_request = request->infer_request; input.height = task->in_frame->height; input.width = task->in_frame->width; - tf_input.oper = TF_GraphOperationByName(tf_model->graph, task->input_name); - if (!tf_input.oper){ + infer_request->tf_input = av_malloc(sizeof(TF_Output)); + infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); + if (!infer_request->tf_input->oper){ av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); return DNN_ERROR; } - tf_input.index = 0; - input_tensor = allocate_input_tensor(&input); - if (!input_tensor){ + infer_request->tf_input->index = 0; + infer_request->input_tensor = allocate_input_tensor(&input); + if (!infer_request->input_tensor){ av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); return DNN_ERROR; } - input.data = (float *)TF_TensorData(input_tensor); + input.data = (float *)TF_TensorData(infer_request->input_tensor); switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: @@ -869,60 +894,52 @@ static DNNReturnType execute_model_tf(Queue *inference_queue) break; } - tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); - if (tf_outputs == NULL) { - TF_DeleteTensor(input_tensor); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); \ + infer_request->tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); + if (infer_request->tf_outputs == NULL) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); return DNN_ERROR; } - output_tensors = av_mallocz_array(task->nb_output, sizeof(*output_tensors)); - if (!output_tensors) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); \ + infer_request->output_tensors = av_mallocz_array(task->nb_output, sizeof(*infer_request->output_tensors)); + if (!infer_request->output_tensors) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); return DNN_ERROR; } for (int i = 0; i < task->nb_output; ++i) { - tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); - if (!tf_outputs[i].oper) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); \ + infer_request->output_tensors[i] = NULL; + infer_request->tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); + if (!infer_request->tf_outputs[i].oper) { + av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); return DNN_ERROR; } - tf_outputs[i].index = 0; + infer_request->tf_outputs[i].index = 0; } TF_SessionRun(tf_model->session, NULL, - &tf_input, &input_tensor, 1, - tf_outputs, output_tensors, task->nb_output, - NULL, 0, NULL, tf_model->status); + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); if (TF_GetCode(tf_model->status) != TF_OK) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; } outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); \ + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); return DNN_ERROR; } for (uint32_t i = 0; i < task->nb_output; ++i) { - outputs[i].height = TF_Dim(output_tensors[i], 1); - outputs[i].width = TF_Dim(output_tensors[i], 2); - outputs[i].channels = TF_Dim(output_tensors[i], 3); - outputs[i].data = TF_TensorData(output_tensors[i]); - outputs[i].dt = TF_TensorType(output_tensors[i]); + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); } switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: @@ -946,30 +963,15 @@ static DNNReturnType execute_model_tf(Queue *inference_queue) tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; default: - for (uint32_t i = 0; i < task->nb_output; ++i) { - if (output_tensors[i]) { - TF_DeleteTensor(output_tensors[i]); - } - } - TF_DeleteTensor(input_tensor); - av_freep(&output_tensors); - av_freep(&tf_outputs); - av_freep(&outputs); + tf_free_request(infer_request); av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); return DNN_ERROR; } - for (uint32_t i = 0; i < task->nb_output; ++i) { - if (output_tensors[i]) { - TF_DeleteTensor(output_tensors[i]); - } - } task->inference_done++; - TF_DeleteTensor(input_tensor); - av_freep(&output_tensors); - av_freep(&tf_outputs); + tf_free_request(infer_request); av_freep(&outputs); - return DNN_SUCCESS; + ff_safe_queue_push_back(tf_model->request_queue, request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } @@ -978,6 +980,7 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * TFModel *tf_model = model->model; TFContext *ctx = &tf_model->ctx; TaskItem task; + TFRequestItem *request; if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { return DNN_ERROR; @@ -991,7 +994,14 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); return DNN_ERROR; } - return execute_model_tf(tf_model->inference_queue); + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + return execute_model_tf(request, tf_model->inference_queue); } void ff_dnn_free_model_tf(DNNModel **model) @@ -1000,6 +1010,14 @@ void ff_dnn_free_model_tf(DNNModel **model) if (*model){ tf_model = (*model)->model; + while (ff_safe_queue_size(tf_model->request_queue) != 0) { + TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + tf_free_request(item->infer_request); + av_freep(&item->infer_request); + av_freep(&item); + } + ff_safe_queue_destroy(tf_model->request_queue); + while (ff_queue_size(tf_model->inference_queue) != 0) { InferenceItem *item = ff_queue_pop_front(tf_model->inference_queue); av_freep(&item); From patchwork Mon Jul 5 10:30:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28782 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4030964ios; Mon, 5 Jul 2021 03:38:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwxH/7D5xC/Fp8hncnerZiS5qgwcmg09Q3lSjWMzH3tSY2oMEsJCq+cmTMnJI9X37YbVza0 X-Received: by 2002:a17:907:3f96:: with SMTP id hr22mr5018607ejc.129.1625481528828; Mon, 05 Jul 2021 03:38:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481528; cv=none; d=google.com; s=arc-20160816; b=j9ebEV2Bu9VMg+DJggNP3oxhKBFv+3/uVztHQMi/ctEcQLx6yzemvxSn6GRnu13/Dv ZlDyA+/e5ARBiCkPOjb4jo8F3p4A/JYoKop6Js2vRAsurB5zudWy/aSczRnO30wwbteW BTRS8zkpmIHuL9A72OoiKcB196qfC71R2/rp60IbwBb0RWB6VNgDM7KXaX0MdM7FJV43 b+DJ/uJFMwHzXCS7JuyiNQlfc2GtpuvYxvl4L/FvNgI//Tp0ckn18WE4AYIIdKDIqq+z kt2x75aJ3WZq/F/651vdwPuwRlsfLjRghZwBSlOhozqKalA2iwUJhG4wA+Sd9eZY/csK YDQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=vtzVYd2UPLLQ1UiVNqgUoMfcg1/joJXT5WsGYApLQHE=; b=kXHyif8Ig0SdVQtBKDkr/GjecPt7M9dPMAxbjZKibtXHsv6MfmbOVMXxHpRg+WMWXd 5cBIyLgUreUeHcrNwd9168QW4UdeXHFJ+x9BYkPvxn1dLi8m5T+1jg+NHu/l/VAhRQKH Ku7wfpAM6fONBKPGaFmMxZY/P09U5GfEiRR/8jeAe9DMzc87StXkIxWagqrY3vyWyY/N QYMyM4cMR4jZEJUMvJ2Hqewv60QiWurFp1QczUkB2zHGW82qgEV3hBgP0J3LCavSFEr4 JbKM9xmLdy1pbWP1n1t5nIRr8rimBgTntug6eQv2fcMWlkkJfA+/XHV+5AKBDC97Gh2O Fd1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=NfepmZnN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d19si11390697edq.167.2021.07.05.03.38.48; Mon, 05 Jul 2021 03:38:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=NfepmZnN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5D2CD68A601; Mon, 5 Jul 2021 13:38:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 92C64689FE2 for ; Mon, 5 Jul 2021 13:38:39 +0300 (EEST) Received: by mail-pl1-f173.google.com with SMTP id i13so9984901plb.10 for ; Mon, 05 Jul 2021 03:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ybc4CdDAuLTEqYrIgvw+9S4WZf/Xl06KIsIW95Wbtzk=; b=NfepmZnNSSVCQhuD/I68sHSIQDJSNJ3ahe1ykXhrfhJ0yIv40qIvC2YnskYHzB63mD fzjm9LFEDQDm52ZAYoPWlpPGK6iPHKepd85NqnYY9lJZWVPBQWeRLR/7KJNcCMyZrVuD wBnFI1jMigZVaJU6la4Kh+FyMxAt94fBFS/ttv2J0ev/89uflLz7kDx0/KjmSN/a+llB 7WFv83hJsKFavpB/K5uFfzNt9O2EHLLGwot+eMzIsYLeSQNm7F4NfW1gxyICYdibXqnK HsOu6eut36CNN1dy3d6vA5Q+Q3QjoA9FXtU1ScY2O0cktVue31IYBH9IdjYL2QqnBSk5 WpeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ybc4CdDAuLTEqYrIgvw+9S4WZf/Xl06KIsIW95Wbtzk=; b=q00XcJQOZbNKAkai74O5gHP2F6Mxnko6Jn9L5R7PBqWui0/iTVQ2jedC4QP6Qom9jd r59dv6k1GAB3q4IjTtxM/zxUuZIz1y/TC/yqZJ8BDAJQsJC1SEfdJFb494DblbAcFa4+ 5qWJaJspkmtKniGL9wIvvqysojBRaGjS+cg4OVI8U4ZdqGsrN43fl0VQpP3hdyMlLlMR ThMxi1+ZF+7Usj0tRDStzl+FaFlnJLsc9LXHFGEVQ1jzdS4ozfCpw+tC7MEGXMDWMo/P ik8qVA2YVKEaceKvU43VcpeLX2sNoaJez0iYeWW9E+d+Tq4FwXjIfnESK9YRoEgOrkBl Onnw== X-Gm-Message-State: AOAM533ql4xePj82SZF3CQ3rt4R7XmcUw0dbsya7f+jiJ+Fhfa3u0qA3 BYzp25a2I5WxjCikm9XQnv26XEb3SOHWXA== X-Received: by 2002:a17:902:ee44:b029:128:fd7a:a84d with SMTP id 4-20020a170902ee44b0290128fd7aa84dmr11779318plo.69.1625481093305; Mon, 05 Jul 2021 03:31:33 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:33 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:56 +0530 Message-Id: <20210705103057.42309-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 4/6] lavfi/dnn_backend_tf: Separate function for filling RequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Q5AHndxMCJvS This commit rearranges the existing code to create separate function for filling request with execution data. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 137 ++++++++++++++++++------------- 1 file changed, 80 insertions(+), 57 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index e8007406c8..7f014d55fa 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -839,20 +839,16 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) -{ - TFModel *tf_model; - TFContext *ctx; - TFInferRequest *infer_request; +static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { + DNNData input; InferenceItem *inference; TaskItem *task; - DNNData input, *outputs; + TFInferRequest *infer_request; + TFContext *ctx = &tf_model->ctx; - inference = ff_queue_pop_front(inference_queue); + inference = ff_queue_pop_front(tf_model->inference_queue); av_assert0(inference); task = inference->task; - tf_model = task->model; - ctx = &tf_model->ctx; request->inference = inference; if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) @@ -916,63 +912,90 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q infer_request->tf_outputs[i].index = 0; } - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } + return DNN_SUCCESS; +} - outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); - if (!outputs) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); +static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) +{ + TFModel *tf_model; + TFContext *ctx; + TFInferRequest *infer_request; + InferenceItem *inference; + TaskItem *task; + DNNData *outputs; + + inference = ff_queue_peek_front(inference_queue); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + + if (task->async) { + avpriv_report_missing_feature(ctx, "Async execution not supported"); return DNN_ERROR; - } + } else { + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { + return DNN_ERROR; + } - for (uint32_t i = 0; i < task->nb_output; ++i) { - outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); - outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); - outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); - outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); - outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); - } - switch (tf_model->model->func_type) { - case DFT_PROCESS_FRAME: - //it only support 1 output if it's frame in & frame out - if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + infer_request = request->infer_request; + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; + } + + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); + if (!outputs) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); + return DNN_ERROR; + } + + for (uint32_t i = 0; i < task->nb_output; ++i) { + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); + } + switch (tf_model->model->func_type) { + case DFT_PROCESS_FRAME: + //it only support 1 output if it's frame in & frame out + if (task->do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + } else { + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + } } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; } - } else { - task->out_frame->width = outputs[0].width; - task->out_frame->height = outputs[0].height; - } - break; - case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { - av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + break; + case DFT_ANALYTICS_DETECT: + if (!tf_model->model->detect_post_proc) { + av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + return DNN_ERROR; + } + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + break; + default: + tf_free_request(infer_request); + + av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); return DNN_ERROR; } - tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); - break; - default: + task->inference_done++; tf_free_request(infer_request); - - av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); - return DNN_ERROR; + av_freep(&outputs); + ff_safe_queue_push_back(tf_model->request_queue, request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } - task->inference_done++; - tf_free_request(infer_request); - av_freep(&outputs); - ff_safe_queue_push_back(tf_model->request_queue, request); - return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) From patchwork Mon Jul 5 10:30:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28783 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4030398ios; Mon, 5 Jul 2021 03:37:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyyzVG9i9RcmIxhLf9Bg/O8Rmo932M4L1k+xgtCNTdtJ4Wc+1YRvK2wyWD4sKjIFZ/E+EpU X-Received: by 2002:a05:6402:100e:: with SMTP id c14mr15047492edu.51.1625481479154; Mon, 05 Jul 2021 03:37:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481479; cv=none; d=google.com; s=arc-20160816; b=ZVywJIxayMVsoRxiYAfhNpdNe4UU+2x1hA2Wn0NMl5by/+9Xq+j97dnaXzouIFxl5s /+U1k7qKb2PbP8Hn7ScnLZfIdO+3pS8OexMkrju27r0TdBUuiLEJbmzCFE2Q8iy4mZ0d tLXiAXxXOODx/orN7hNzbO6D1192UAO4YH854ImVTQx6F1H0JIKOnDBbTBOkZxmm4pQB 0wzPpj9KzPxYrW7F1cbUvZ7FITrzTO5Up2us0GdoQz+8T7H9TnBytOlc6+59CRHUbVLI WI9kQbY9ZOFS5VgXjt2CJLLAkAsJ49x6fyOJX3EOg3i/UyDBSqgeSbSdNTKrttSVOHHk oVFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=x38SAB8qNs0I8LPhck1NLcjeEasscEvX9Ko16iEwz9Q=; b=eWt3nxJWcYP1Ungs/kEX9hJ4XounKx8oZXMnWoaV1SKHjTI+QovFn0LYTS3PQRJnEi gMFp/0A7eOv0+vMX1D0sCs/BDp1ST2slhwYMdbtEArRuw7iqMnb3eLVeZL8ESd4rOdxx MdKuRmylJB9fia5qcDuSlyrJIm9SBbcwqZgz/EoGDXd3AlG8laMmqKRi1STFspHwx3g+ 3eqZ8a9rxl9BRlYgyUT5tudLe4Jk1CyD34q5tnNGlqhu4yxZTxXUrjW+6F7OgtTbP9Ui urQFF3366rDuw5e1fII16/NwC7CbYKnnovLHkNGMHpjzfFIdAcdykcH0erB6eMuBVfdV kFIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dsa6RbDh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id em21si10942703ejc.10.2021.07.05.03.37.58; Mon, 05 Jul 2021 03:37:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dsa6RbDh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 363AA689FE2; Mon, 5 Jul 2021 13:37:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0D96A689FE2 for ; Mon, 5 Jul 2021 13:37:50 +0300 (EEST) Received: by mail-qv1-f46.google.com with SMTP id j14so8051725qvu.6 for ; Mon, 05 Jul 2021 03:37:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l0wEvqzCilkIfDqbEMhMVi6SiQ5VU343UqkiB5mZkSk=; b=dsa6RbDhSaX+YihWZPY856M9TBs0folv3AVBGKpj5SyoDqV7FvMhE+r6VuWHOORKgX Zi3hHHgvh9JqxtupQTXGQ0WOKRqBU2RPNK4im20cJ4OcE4xEdsD384RRNTZBnflV2pe5 YfpHD2kPjxve8nKWOoE66VaZhBv35D5X//Oi68Z1beRAGYZ1OBAq0Mii/25UD2jTGRdQ twtniCuEwXa+6NyBmDb50zdMAxVYsJiwnlIk5hn3YqkOTrkf6ZRRFb1GBMH4VAa29OHp A2rsJ45PeatzzHq49ERWxagN5swwiIeTt8zez+POWEAkQU0D61/1B1u959HtbzQurcEC rfsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l0wEvqzCilkIfDqbEMhMVi6SiQ5VU343UqkiB5mZkSk=; b=NmnV8RBXLLWTozlaAi8phwyQpFgRR8WhnNXNeXtfDAEBJUw0bjDhSRDTw1jRfRTRte EQ4/rfYJO+hZR8KEu2HtxskU/YQKOPXMc2LG8qtpQOm/IrL1zSvCq813qO0mVPYbLP1o 1fMajSEv9iAmNfDznC3EGlwjY5xq+UmZolqmViq63wiyjg+95N8Vbf0qOO/5YJNXw7y6 1aopCvRWxoPzRXA+ZNdR9RsbhST/fX4jk6r7Z9hdd/d3q17/SIb+92HDiqxoP5ztIS/g CeJNx39zqHFQjeQaAMr3Wsq+7KIV+S5rvnXtB9tlUfEvFAikXgMn4pYxQHubeXz0ec6L 18Lg== X-Gm-Message-State: AOAM530axAWTPexAhQtABMZIVuXie4q8+6AtBw5neTrVdsSITE/rHDIL ta4NKa0x771jo9yULWYhFRfbNZ8T+y54pg== X-Received: by 2002:a17:902:ba89:b029:11e:7a87:207 with SMTP id k9-20020a170902ba89b029011e7a870207mr11853961pls.81.1625481095175; Mon, 05 Jul 2021 03:31:35 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:34 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:57 +0530 Message-Id: <20210705103057.42309-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 5/6] lavfi/dnn_backend_tf: Separate function for Completion Callback X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mGYQI2Jj43Ji This commit rearranges the existing code to create a separate function for the completion callback in execute_model_tf. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 109 +++++++++++++++++-------------- 1 file changed, 61 insertions(+), 48 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 7f014d55fa..6664d7194b 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -915,6 +915,65 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *reque return DNN_SUCCESS; } +static void infer_completion_callback(void *args) { + TFRequestItem *request = args; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + DNNData *outputs; + TFInferRequest *infer_request = request->infer_request; + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; + + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); + if (!outputs) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); + goto err; + } + + for (uint32_t i = 0; i < task->nb_output; ++i) { + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); + } + switch (tf_model->model->func_type) { + case DFT_PROCESS_FRAME: + //it only support 1 output if it's frame in & frame out + if (task->do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + } else { + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + } + } else { + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; + } + break; + case DFT_ANALYTICS_DETECT: + if (!tf_model->model->detect_post_proc) { + av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + return; + } + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + break; + default: + av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); + goto err; + } + task->inference_done++; +err: + tf_free_request(infer_request); + av_freep(&outputs); + + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_freep(&request->infer_request); + av_freep(&request); + av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); + } +} + static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) { TFModel *tf_model; @@ -922,7 +981,6 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; - DNNData *outputs; inference = ff_queue_peek_front(inference_queue); task = inference->task; @@ -944,56 +1002,11 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q task->nb_output, NULL, 0, NULL, tf_model->status); if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } - - outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); - if (!outputs) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); - return DNN_ERROR; - } - - for (uint32_t i = 0; i < task->nb_output; ++i) { - outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); - outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); - outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); - outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); - outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); - } - switch (tf_model->model->func_type) { - case DFT_PROCESS_FRAME: - //it only support 1 output if it's frame in & frame out - if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); - } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); - } - } else { - task->out_frame->width = outputs[0].width; - task->out_frame->height = outputs[0].height; - } - break; - case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { - av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); - return DNN_ERROR; - } - tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); - break; - default: tf_free_request(infer_request); - - av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); return DNN_ERROR; } - task->inference_done++; - tf_free_request(infer_request); - av_freep(&outputs); - ff_safe_queue_push_back(tf_model->request_queue, request); + infer_completion_callback(request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } } From patchwork Mon Jul 5 10:30:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28780 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4026084ios; Mon, 5 Jul 2021 03:31:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyMRFhVo8xL789x5Xoo752ISWkb2WHi0MpTFW+KG27HcIdOe19UjDix1DlxeM8lMv6F6DJF X-Received: by 2002:a05:6402:c10:: with SMTP id co16mr14928065edb.192.1625481112286; Mon, 05 Jul 2021 03:31:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481112; cv=none; d=google.com; s=arc-20160816; b=IBnoKjqoNWyES0t/nG32uxtqhXlyDVjYu4kQyqvm1nagRUSDQN7/MEZZfxHTsDsWcM ICvAErcbrZSB8rXTioAZiyhGGo/yvy5TKojX695+xpL93mXDYvSRqrIY2wJMKRUJadzo +GElsNADVVm2uPumk50DhZ1qLrVqxLuxM5dwPqve3/hSDF2EapLpe94Jk/zE6BoHKWXr 8SlIpZqBa488AeqLPs3DmCtZ95e2CffO5RyJk5GZW0XCy73rEmfB7pcsgVAqaPLozCTI OtOZzhfKXL56XTqm2fV7N8yzMEaHDsAou6DmbeD8izXlumVHLUEDyAK88Tz3q+4KPMUy It5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=mIMhwtjHiVHbpK0BhayJoeUOgQfwKS9uvXqCWlZVIZg=; b=fBwL85JYml8MgLOpcb8YxgNY5bsQj7ZD/8UDu+FO1KVjzJovwrbnb+MT3xO9rkGe7/ T25o4Lo3pFuSsVPcbxnox8Pf9SA+2aD8suUUOtt8UBtSGt746/2zJDbT0rK+tJiqB8ts b5LpQEKbRqHihWGk92Tnpz7VUY7+u+Ksvf7aA9uy6PJrf43FzWMhg+qOJ2Gw+FR9jS99 tabdrzqVFgt9dWvaqJcjwsFjjSQGlrws7IQJ3Akj1Vc/a2VwURmOocnpuinsuYJ0dIiz ONEMXDu928c3CaiYyQYYbTPOw4Rggvx53URxBSTJMxIIV1mYuaSw++GrnEmFYOW8l5f7 ytMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=IhmCiI1O; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 23si12041251ejg.6.2021.07.05.03.31.51; Mon, 05 Jul 2021 03:31:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=IhmCiI1O; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 94C4E68A5EC; Mon, 5 Jul 2021 13:31:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B9F5D68A65A for ; Mon, 5 Jul 2021 13:31:39 +0300 (EEST) Received: by mail-pl1-f175.google.com with SMTP id i13so9976046plb.10 for ; Mon, 05 Jul 2021 03:31:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OLi0SAuWaDS7Gdtroll0XRBFMltoJ2tkXwM36oYVMKs=; b=IhmCiI1OBTbV2g10elGRnsPISd1T5GiGRl/iNo9EerlV4MqNSCW24z80gcoiLJuh4d yFwyHhC/i7PBEJg2vvRsf4lVRyoyd5o0DRLD3Me/E9VU4ktofddjIsPx6o9wWYuUmARg ne3+3yw8dnqq6uqEcBlsSdoQeKoxhLEkVp5Kz2MRqZRnJy5fOp1GZEsx+Jh1agHyqElp IsCvpXe3jOHqLvyyhFShWtRDov7MtZGOohjMGNA8l4iAlEuNh76362tz/gIf5OuSPPX7 mN0CpvS2Yqf2FbkKPDpTHo47KhmwVV3fGhfsaJSo+kFEchQwWXUHQri11WblxdcRKFVC A35A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OLi0SAuWaDS7Gdtroll0XRBFMltoJ2tkXwM36oYVMKs=; b=BminwVeCMFNLWDXLejnlLwLkwtUEbGEmCP1NU7HjBrfQpMp+0FW5i2cRqh/njr2V8t xOLlCCa9QcKFG+5+Rjj6ZWO6ESXXoD1IqGDM7CGivtjOBZyqVmLMISdLG4KQmtCJnfjH tan5lHPh9+q6FDvipO9SPAQ0nyS/q8X7ooEHYSavHbc8WG/yQPWodqidrJVbVGoz9hZR kr8ykdDYh2xq6Jnt8vRpYDEEvhYtHFS5oElMVu5Swn+u0+inXWijmpy7voauF+viabAg jzE1Xv8S0jAojKpjJVVuhs60BsYfW7A6e6dU2lkOSWJ181Lfmf2e+gGJT70zuhmfoaAM DEGg== X-Gm-Message-State: AOAM532/rsXfsbDD4akTiQ7VZa+ouwVgwSzfhxs/NPmQ5y+z4LymD63t een9fQLwCiqQ6oqiR/Gb2atu2TcXDbHVmA== X-Received: by 2002:a17:90a:f68f:: with SMTP id cl15mr14350137pjb.219.1625481097190; Mon, 05 Jul 2021 03:31:37 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:36 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:58 +0530 Message-Id: <20210705103057.42309-6-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 6/6] lavfi/dnn_backend_tf: Error Handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: a7WSC5fJXUB9 This commit adds handling for cases where an error may occur, clearing the allocated memory resources. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 80 ++++++++++++++++++++++++-------- 1 file changed, 60 insertions(+), 20 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 6664d7194b..1d4ca516c1 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -127,14 +127,18 @@ static TFInferRequest *tf_create_inference_request(void) static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; InferenceItem *inference = av_malloc(sizeof(*inference)); if (!inference) { + av_log(ctx, AV_LOG_ERROR, "Unable to allocate space for InferenceItem\n"); return DNN_ERROR; } task->inference_todo = 1; task->inference_done = 0; inference->task = task; if (ff_queue_push_back(inference_queue, inference) < 0) { + av_log(ctx, AV_LOG_ERROR, "Failed to push back inference_queue.\n"); av_freep(&inference); return DNN_ERROR; } @@ -245,14 +249,15 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } out_frame = av_frame_alloc(); if (!out_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } in_frame->width = input_width; @@ -269,19 +274,22 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } request = ff_safe_queue_pop_front(tf_model->request_queue); if (!request) { av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } ret = execute_model_tf(request, tf_model->inference_queue); *output_width = out_frame->width; *output_height = out_frame->height; +err: av_frame_free(&out_frame); av_frame_free(&in_frame); return ret; @@ -802,17 +810,12 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_opt_set_defaults(ctx); if (av_opt_set_from_string(ctx, options, NULL, "=", "&") < 0) { av_log(ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - av_freep(&tf_model); - av_freep(&model); - return NULL; + goto err; } if (load_tf_model(tf_model, model_filename) != DNN_SUCCESS){ if (load_native_model(tf_model, model_filename) != DNN_SUCCESS){ - av_freep(&tf_model); - av_freep(&model); - - return NULL; + goto err; } } @@ -821,14 +824,34 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } tf_model->request_queue = ff_safe_queue_create(); + if (!tf_model->request_queue) { + goto err; + } for (int i = 0; i < ctx->options.nireq; i++) { TFRequestItem *item = av_mallocz(sizeof(*item)); + if (!item) { + goto err; + } item->infer_request = tf_create_inference_request(); - ff_safe_queue_push_back(tf_model->request_queue, item); + if (!item->infer_request) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for TensorFlow inference request\n"); + av_freep(&item); + goto err; + } + + if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { + av_freep(&item->infer_request); + av_freep(&item); + goto err; + } } tf_model->inference_queue = ff_queue_create(); + if (!tf_model->inference_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -837,6 +860,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ model->func_type = func_type; return model; +err: + ff_dnn_free_model_tf(&model); + return NULL; } static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { @@ -851,24 +877,31 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *reque task = inference->task; request->inference = inference; - if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) - return DNN_ERROR; + if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) { + goto err; + } infer_request = request->infer_request; input.height = task->in_frame->height; input.width = task->in_frame->width; infer_request->tf_input = av_malloc(sizeof(TF_Output)); + if (!infer_request->tf_input) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); + goto err; + } + infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); if (!infer_request->tf_input->oper){ av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); - return DNN_ERROR; + goto err; } infer_request->tf_input->index = 0; + infer_request->input_tensor = allocate_input_tensor(&input); if (!infer_request->input_tensor){ av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); - return DNN_ERROR; + goto err; } input.data = (float *)TF_TensorData(infer_request->input_tensor); @@ -893,13 +926,13 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *reque infer_request->tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); if (infer_request->tf_outputs == NULL) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); - return DNN_ERROR; + goto err; } infer_request->output_tensors = av_mallocz_array(task->nb_output, sizeof(*infer_request->output_tensors)); if (!infer_request->output_tensors) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); - return DNN_ERROR; + goto err; } for (int i = 0; i < task->nb_output; ++i) { @@ -907,12 +940,15 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *reque infer_request->tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); if (!infer_request->tf_outputs[i].oper) { av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); - return DNN_ERROR; + goto err; } infer_request->tf_outputs[i].index = 0; } return DNN_SUCCESS; +err: + tf_free_request(infer_request); + return DNN_ERROR; } static void infer_completion_callback(void *args) { @@ -983,6 +1019,10 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q TaskItem *task; inference = ff_queue_peek_front(inference_queue); + if (!inference) { + av_log(NULL, AV_LOG_ERROR, "Failed to get inference item\n"); + return DNN_ERROR; + } task = inference->task; tf_model = task->model; ctx = &tf_model->ctx;