From patchwork Fri May 28 09:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27962 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368682iof; Fri, 28 May 2021 02:26:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzJx3eYpYdPsZE/qj2rMV0Ixwb8NKRuPNiQSyy7EdynGP0+QejkWtWQcUqsep5/c7azaNtp X-Received: by 2002:a17:906:8303:: with SMTP id j3mr5341550ejx.403.1622194009198; Fri, 28 May 2021 02:26:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622194009; cv=none; d=google.com; s=arc-20160816; b=xOKQyaWvhR7H5cQn64aekILskpWMoz2RPIu3q16lHDz6XEaecx5FJKfGEtrJa5oqtm RYyIYOgLtncBXX2xPKY1Vbg9aQd/pJrhFMkYgD7Mr7+gA9hqidikBcwTHFq+SwzjUA5L txm8SP1GAKdS/Fld/1+9KPXtWDenKac9Dp2h7Z6KfSEQvSeTuzCk3sQQxV1s1vhFXMFx Im1jySC93Xp3MaTJEVnaBM74IfG14babQq1wBnjoV8/vVAT3oHfGX+vKpodi8BHeD7w4 5VbXikfZq0FUl7K34zr7mBi3Gnp2cSQwDQJBUEYo9Xq7odhSaiZaL0s0bUskAXhH0+Ya mcOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=H5EOkytwAUDYgiYxYe0DDBcOY/67AscbbghqeOLsPZM=; b=PA+NVcb9ygN3i0pv8LBvejgOPy7+0hKlUlwAWaJvnBrngeSqaOL2TJrm6oMlaHykCf OPBdEdr6iECUUPJW0UZXQS0u/S9v7903BPymnzXJeFJCAVBi48SNCDW9lU80SkETj1Q8 gKD9E0+J8lvaI3YUZR8p5JeT+0s3f28bmu8MbeT2+S3akrjSzqfnKIamgO8ee/pp1Fn2 tddRBZCOXg1Sh6M/gkZz7so05vfXA3dYe5IkS1fvN5p2Us+DQRyRnmZ7Z7TOpa6TUxg2 2vLQuaFc/dirkVTlZm3Qr6qennPPxAUFGGTZZuOVKFJyExOkfnJs4/+5GUn3FNA8tN5V iWWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=lpSJ+LBz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id he12si4464831ejc.636.2021.05.28.02.26.48; Fri, 28 May 2021 02:26:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=lpSJ+LBz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 054BD68A08E; Fri, 28 May 2021 12:26:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C074A689EA6 for ; Fri, 28 May 2021 12:26:00 +0300 (EEST) Received: by mail-pj1-f42.google.com with SMTP id f3-20020a17090a4a83b02901619627235bso3344pjh.1 for ; Fri, 28 May 2021 02:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uHkRtYZa2QhT8vt5BjboRgyhSUN5OXGfLaCclbnxZE0=; b=lpSJ+LBzwL1UAXW7MYTPyzDX5bdCOSkcCKpAgbFO/h0dUvUswhR8v6+MnT1EFd2XjK 1jPt58qvLiRBYAHIFneoc/H16MZmrqME/uHU4m5hdhhkqYjyPdAh59/PQb6NrlO2yhas CbhlTq0slIBRiI9GDM3BOOdkkjF/MMImO5092i+FF1skfWEmfnE+Chalaq/Q2t0Gx0xV 5nZUz3WUSy8rK52Lykh3RMZDEiIe51GUZXGc07RByc6BRzI1b7mehrVlkjFh5qR5wFTG Vh5nSULmhJsh6S2PvEdSmhGsFOBDF1EQ0+MYbxCrGUvV41ZklNGnQZSPXXFd0qZA+RZN K0IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uHkRtYZa2QhT8vt5BjboRgyhSUN5OXGfLaCclbnxZE0=; b=kX4+XVpUHwz1teHILUsYwawLJ4uDtLWJizSFiJJ0s3EtfJ/ikETa1Q5jqiV+u7LTlx H9L+BClPRHQq1UM0TgNeirXQLs6JI8nrtK8RcmfVK4tURzNpzDJwxuFFLDbSnKvkgval 0zBnjvLNmpoVEIaEYWGoYTA37PrFoaRDUT2aqSzOFaJ3XrLfy21ktKDl/KK84qHJaGJC WmziwFVhVl/bFTHwSEtBhQ3fdvsVxB86lhwLkY6ZQksJz8p/K5tBttcq44rfhWrauy/+ kmf6prTrqU8GOsONbvWPdfNbavWgyarCmKWLGfVL0xBIEKMq/BX8IOkAr7bHdWhdelw9 3naA== X-Gm-Message-State: AOAM533VzAIGBxbXkPicvZlcmS9f0wrD/OhbDj+c6/tYQfup+jnYx8mL PKMe7lq0NPoYqGR8DV6qZhnZRuoYbZFcNA== X-Received: by 2002:a17:902:724c:b029:ef:571f:8894 with SMTP id c12-20020a170902724cb02900ef571f8894mr7222607pll.49.1622193958981; Fri, 28 May 2021 02:25:58 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:58 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:51 +0530 Message-Id: <20210528092454.31874-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 07/10] lavfi/dnn_backend_tf: Separate function for filling RequestItem and callback X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Tk9CsIegggka This commit rearranges the existing code to create two separate functions for filling request with execution data and the completion callback. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 81 ++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 24 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 793b108e55..5d34da5db1 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -826,20 +826,16 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) -{ - TFModel *tf_model; - TFContext *ctx; - tf_infer_request *infer_request; +static DNNReturnType fill_model_input_tf(TFModel *tf_model, RequestItem *request) { + DNNData input; InferenceItem *inference; TaskItem *task; - DNNData input, *outputs; + tf_infer_request *infer_request; + TFContext *ctx = &tf_model->ctx; - inference = ff_queue_pop_front(inference_queue); + inference = ff_queue_pop_front(tf_model->inference_queue); av_assert0(inference); task = inference->task; - tf_model = task->model; - ctx = &tf_model->ctx; request->inference = inference; if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) @@ -852,7 +848,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que infer_request->tf_input = av_malloc(sizeof(TF_Output)); infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); if (!infer_request->tf_input->oper){ - av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name); + av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); return DNN_ERROR; } infer_request->tf_input->index = 0; @@ -902,22 +898,23 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que infer_request->tf_outputs[i].index = 0; } - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } + return DNN_SUCCESS; +} + +static void infer_completion_callback(void *args) { + RequestItem *request = args; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + DNNData *outputs; + tf_infer_request *infer_request = request->infer_request; + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { tf_free_request(infer_request); av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); - return DNN_ERROR; + return; } for (uint32_t i = 0; i < task->nb_output; ++i) { @@ -944,7 +941,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que case DFT_ANALYTICS_DETECT: if (!tf_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); - return DNN_ERROR; + return; } tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; @@ -955,7 +952,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que } } av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); - return DNN_ERROR; + return; } for (uint32_t i = 0; i < task->nb_output; ++i) { if (infer_request->output_tensors[i]) { @@ -966,7 +963,43 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que tf_free_request(infer_request); av_freep(&outputs); ff_safe_queue_push_back(tf_model->request_queue, request); - return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; +} + +static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) +{ + TFModel *tf_model; + TFContext *ctx; + tf_infer_request *infer_request; + InferenceItem *inference; + TaskItem *task; + + inference = ff_queue_peek_front(inference_queue); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + + if (task->async) { + avpriv_report_missing_feature(ctx, "Async execution not supported"); + return DNN_ERROR; + } else { + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { + return DNN_ERROR; + } + + infer_request = request->infer_request; + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; + } + infer_completion_callback(request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; + } } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params)