From patchwork Mon Jul 5 10:30:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28783 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4030398ios; Mon, 5 Jul 2021 03:37:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyyzVG9i9RcmIxhLf9Bg/O8Rmo932M4L1k+xgtCNTdtJ4Wc+1YRvK2wyWD4sKjIFZ/E+EpU X-Received: by 2002:a05:6402:100e:: with SMTP id c14mr15047492edu.51.1625481479154; Mon, 05 Jul 2021 03:37:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481479; cv=none; d=google.com; s=arc-20160816; b=ZVywJIxayMVsoRxiYAfhNpdNe4UU+2x1hA2Wn0NMl5by/+9Xq+j97dnaXzouIFxl5s /+U1k7qKb2PbP8Hn7ScnLZfIdO+3pS8OexMkrju27r0TdBUuiLEJbmzCFE2Q8iy4mZ0d tLXiAXxXOODx/orN7hNzbO6D1192UAO4YH854ImVTQx6F1H0JIKOnDBbTBOkZxmm4pQB 0wzPpj9KzPxYrW7F1cbUvZ7FITrzTO5Up2us0GdoQz+8T7H9TnBytOlc6+59CRHUbVLI WI9kQbY9ZOFS5VgXjt2CJLLAkAsJ49x6fyOJX3EOg3i/UyDBSqgeSbSdNTKrttSVOHHk oVFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=x38SAB8qNs0I8LPhck1NLcjeEasscEvX9Ko16iEwz9Q=; b=eWt3nxJWcYP1Ungs/kEX9hJ4XounKx8oZXMnWoaV1SKHjTI+QovFn0LYTS3PQRJnEi gMFp/0A7eOv0+vMX1D0sCs/BDp1ST2slhwYMdbtEArRuw7iqMnb3eLVeZL8ESd4rOdxx MdKuRmylJB9fia5qcDuSlyrJIm9SBbcwqZgz/EoGDXd3AlG8laMmqKRi1STFspHwx3g+ 3eqZ8a9rxl9BRlYgyUT5tudLe4Jk1CyD34q5tnNGlqhu4yxZTxXUrjW+6F7OgtTbP9Ui urQFF3366rDuw5e1fII16/NwC7CbYKnnovLHkNGMHpjzfFIdAcdykcH0erB6eMuBVfdV kFIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dsa6RbDh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id em21si10942703ejc.10.2021.07.05.03.37.58; Mon, 05 Jul 2021 03:37:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dsa6RbDh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 363AA689FE2; Mon, 5 Jul 2021 13:37:56 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0D96A689FE2 for ; Mon, 5 Jul 2021 13:37:50 +0300 (EEST) Received: by mail-qv1-f46.google.com with SMTP id j14so8051725qvu.6 for ; Mon, 05 Jul 2021 03:37:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l0wEvqzCilkIfDqbEMhMVi6SiQ5VU343UqkiB5mZkSk=; b=dsa6RbDhSaX+YihWZPY856M9TBs0folv3AVBGKpj5SyoDqV7FvMhE+r6VuWHOORKgX Zi3hHHgvh9JqxtupQTXGQ0WOKRqBU2RPNK4im20cJ4OcE4xEdsD384RRNTZBnflV2pe5 YfpHD2kPjxve8nKWOoE66VaZhBv35D5X//Oi68Z1beRAGYZ1OBAq0Mii/25UD2jTGRdQ twtniCuEwXa+6NyBmDb50zdMAxVYsJiwnlIk5hn3YqkOTrkf6ZRRFb1GBMH4VAa29OHp A2rsJ45PeatzzHq49ERWxagN5swwiIeTt8zez+POWEAkQU0D61/1B1u959HtbzQurcEC rfsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l0wEvqzCilkIfDqbEMhMVi6SiQ5VU343UqkiB5mZkSk=; b=NmnV8RBXLLWTozlaAi8phwyQpFgRR8WhnNXNeXtfDAEBJUw0bjDhSRDTw1jRfRTRte EQ4/rfYJO+hZR8KEu2HtxskU/YQKOPXMc2LG8qtpQOm/IrL1zSvCq813qO0mVPYbLP1o 1fMajSEv9iAmNfDznC3EGlwjY5xq+UmZolqmViq63wiyjg+95N8Vbf0qOO/5YJNXw7y6 1aopCvRWxoPzRXA+ZNdR9RsbhST/fX4jk6r7Z9hdd/d3q17/SIb+92HDiqxoP5ztIS/g CeJNx39zqHFQjeQaAMr3Wsq+7KIV+S5rvnXtB9tlUfEvFAikXgMn4pYxQHubeXz0ec6L 18Lg== X-Gm-Message-State: AOAM530axAWTPexAhQtABMZIVuXie4q8+6AtBw5neTrVdsSITE/rHDIL ta4NKa0x771jo9yULWYhFRfbNZ8T+y54pg== X-Received: by 2002:a17:902:ba89:b029:11e:7a87:207 with SMTP id k9-20020a170902ba89b029011e7a870207mr11853961pls.81.1625481095175; Mon, 05 Jul 2021 03:31:35 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:34 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:57 +0530 Message-Id: <20210705103057.42309-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 5/6] lavfi/dnn_backend_tf: Separate function for Completion Callback X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: mGYQI2Jj43Ji This commit rearranges the existing code to create a separate function for the completion callback in execute_model_tf. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 109 +++++++++++++++++-------------- 1 file changed, 61 insertions(+), 48 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 7f014d55fa..6664d7194b 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -915,6 +915,65 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *reque return DNN_SUCCESS; } +static void infer_completion_callback(void *args) { + TFRequestItem *request = args; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + DNNData *outputs; + TFInferRequest *infer_request = request->infer_request; + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; + + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); + if (!outputs) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); + goto err; + } + + for (uint32_t i = 0; i < task->nb_output; ++i) { + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); + } + switch (tf_model->model->func_type) { + case DFT_PROCESS_FRAME: + //it only support 1 output if it's frame in & frame out + if (task->do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + } else { + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + } + } else { + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; + } + break; + case DFT_ANALYTICS_DETECT: + if (!tf_model->model->detect_post_proc) { + av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + return; + } + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + break; + default: + av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); + goto err; + } + task->inference_done++; +err: + tf_free_request(infer_request); + av_freep(&outputs); + + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_freep(&request->infer_request); + av_freep(&request); + av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); + } +} + static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) { TFModel *tf_model; @@ -922,7 +981,6 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; - DNNData *outputs; inference = ff_queue_peek_front(inference_queue); task = inference->task; @@ -944,56 +1002,11 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q task->nb_output, NULL, 0, NULL, tf_model->status); if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } - - outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); - if (!outputs) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); - return DNN_ERROR; - } - - for (uint32_t i = 0; i < task->nb_output; ++i) { - outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); - outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); - outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); - outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); - outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); - } - switch (tf_model->model->func_type) { - case DFT_PROCESS_FRAME: - //it only support 1 output if it's frame in & frame out - if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); - } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); - } - } else { - task->out_frame->width = outputs[0].width; - task->out_frame->height = outputs[0].height; - } - break; - case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { - av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); - return DNN_ERROR; - } - tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); - break; - default: tf_free_request(infer_request); - - av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); return DNN_ERROR; } - task->inference_done++; - tf_free_request(infer_request); - av_freep(&outputs); - ff_safe_queue_push_back(tf_model->request_queue, request); + infer_completion_callback(request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } }