From patchwork Mon Jul 5 10:30:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 28782 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp4030964ios; Mon, 5 Jul 2021 03:38:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwxH/7D5xC/Fp8hncnerZiS5qgwcmg09Q3lSjWMzH3tSY2oMEsJCq+cmTMnJI9X37YbVza0 X-Received: by 2002:a17:907:3f96:: with SMTP id hr22mr5018607ejc.129.1625481528828; Mon, 05 Jul 2021 03:38:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625481528; cv=none; d=google.com; s=arc-20160816; b=j9ebEV2Bu9VMg+DJggNP3oxhKBFv+3/uVztHQMi/ctEcQLx6yzemvxSn6GRnu13/Dv ZlDyA+/e5ARBiCkPOjb4jo8F3p4A/JYoKop6Js2vRAsurB5zudWy/aSczRnO30wwbteW BTRS8zkpmIHuL9A72OoiKcB196qfC71R2/rp60IbwBb0RWB6VNgDM7KXaX0MdM7FJV43 b+DJ/uJFMwHzXCS7JuyiNQlfc2GtpuvYxvl4L/FvNgI//Tp0ckn18WE4AYIIdKDIqq+z kt2x75aJ3WZq/F/651vdwPuwRlsfLjRghZwBSlOhozqKalA2iwUJhG4wA+Sd9eZY/csK YDQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=vtzVYd2UPLLQ1UiVNqgUoMfcg1/joJXT5WsGYApLQHE=; b=kXHyif8Ig0SdVQtBKDkr/GjecPt7M9dPMAxbjZKibtXHsv6MfmbOVMXxHpRg+WMWXd 5cBIyLgUreUeHcrNwd9168QW4UdeXHFJ+x9BYkPvxn1dLi8m5T+1jg+NHu/l/VAhRQKH Ku7wfpAM6fONBKPGaFmMxZY/P09U5GfEiRR/8jeAe9DMzc87StXkIxWagqrY3vyWyY/N QYMyM4cMR4jZEJUMvJ2Hqewv60QiWurFp1QczUkB2zHGW82qgEV3hBgP0J3LCavSFEr4 JbKM9xmLdy1pbWP1n1t5nIRr8rimBgTntug6eQv2fcMWlkkJfA+/XHV+5AKBDC97Gh2O Fd1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=NfepmZnN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d19si11390697edq.167.2021.07.05.03.38.48; Mon, 05 Jul 2021 03:38:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=NfepmZnN; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5D2CD68A601; Mon, 5 Jul 2021 13:38:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 92C64689FE2 for ; Mon, 5 Jul 2021 13:38:39 +0300 (EEST) Received: by mail-pl1-f173.google.com with SMTP id i13so9984901plb.10 for ; Mon, 05 Jul 2021 03:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ybc4CdDAuLTEqYrIgvw+9S4WZf/Xl06KIsIW95Wbtzk=; b=NfepmZnNSSVCQhuD/I68sHSIQDJSNJ3ahe1ykXhrfhJ0yIv40qIvC2YnskYHzB63mD fzjm9LFEDQDm52ZAYoPWlpPGK6iPHKepd85NqnYY9lJZWVPBQWeRLR/7KJNcCMyZrVuD wBnFI1jMigZVaJU6la4Kh+FyMxAt94fBFS/ttv2J0ev/89uflLz7kDx0/KjmSN/a+llB 7WFv83hJsKFavpB/K5uFfzNt9O2EHLLGwot+eMzIsYLeSQNm7F4NfW1gxyICYdibXqnK HsOu6eut36CNN1dy3d6vA5Q+Q3QjoA9FXtU1ScY2O0cktVue31IYBH9IdjYL2QqnBSk5 WpeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ybc4CdDAuLTEqYrIgvw+9S4WZf/Xl06KIsIW95Wbtzk=; b=q00XcJQOZbNKAkai74O5gHP2F6Mxnko6Jn9L5R7PBqWui0/iTVQ2jedC4QP6Qom9jd r59dv6k1GAB3q4IjTtxM/zxUuZIz1y/TC/yqZJ8BDAJQsJC1SEfdJFb494DblbAcFa4+ 5qWJaJspkmtKniGL9wIvvqysojBRaGjS+cg4OVI8U4ZdqGsrN43fl0VQpP3hdyMlLlMR ThMxi1+ZF+7Usj0tRDStzl+FaFlnJLsc9LXHFGEVQ1jzdS4ozfCpw+tC7MEGXMDWMo/P ik8qVA2YVKEaceKvU43VcpeLX2sNoaJez0iYeWW9E+d+Tq4FwXjIfnESK9YRoEgOrkBl Onnw== X-Gm-Message-State: AOAM533ql4xePj82SZF3CQ3rt4R7XmcUw0dbsya7f+jiJ+Fhfa3u0qA3 BYzp25a2I5WxjCikm9XQnv26XEb3SOHWXA== X-Received: by 2002:a17:902:ee44:b029:128:fd7a:a84d with SMTP id 4-20020a170902ee44b0290128fd7aa84dmr11779318plo.69.1625481093305; Mon, 05 Jul 2021 03:31:33 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.113]) by smtp.googlemail.com with ESMTPSA id cx5sm13174242pjb.1.2021.07.05.03.31.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Jul 2021 03:31:33 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Mon, 5 Jul 2021 16:00:56 +0530 Message-Id: <20210705103057.42309-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210705103057.42309-1-shubhanshu.e01@gmail.com> References: <20210705103057.42309-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 4/6] lavfi/dnn_backend_tf: Separate function for filling RequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Q5AHndxMCJvS This commit rearranges the existing code to create separate function for filling request with execution data. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 137 ++++++++++++++++++------------- 1 file changed, 80 insertions(+), 57 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index e8007406c8..7f014d55fa 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -839,20 +839,16 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) -{ - TFModel *tf_model; - TFContext *ctx; - TFInferRequest *infer_request; +static DNNReturnType fill_model_input_tf(TFModel *tf_model, TFRequestItem *request) { + DNNData input; InferenceItem *inference; TaskItem *task; - DNNData input, *outputs; + TFInferRequest *infer_request; + TFContext *ctx = &tf_model->ctx; - inference = ff_queue_pop_front(inference_queue); + inference = ff_queue_pop_front(tf_model->inference_queue); av_assert0(inference); task = inference->task; - tf_model = task->model; - ctx = &tf_model->ctx; request->inference = inference; if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) @@ -916,63 +912,90 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q infer_request->tf_outputs[i].index = 0; } - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } + return DNN_SUCCESS; +} - outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); - if (!outputs) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); +static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue) +{ + TFModel *tf_model; + TFContext *ctx; + TFInferRequest *infer_request; + InferenceItem *inference; + TaskItem *task; + DNNData *outputs; + + inference = ff_queue_peek_front(inference_queue); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + + if (task->async) { + avpriv_report_missing_feature(ctx, "Async execution not supported"); return DNN_ERROR; - } + } else { + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { + return DNN_ERROR; + } - for (uint32_t i = 0; i < task->nb_output; ++i) { - outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); - outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); - outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); - outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); - outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); - } - switch (tf_model->model->func_type) { - case DFT_PROCESS_FRAME: - //it only support 1 output if it's frame in & frame out - if (task->do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + infer_request = request->infer_request; + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; + } + + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); + if (!outputs) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); + return DNN_ERROR; + } + + for (uint32_t i = 0; i < task->nb_output; ++i) { + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); + } + switch (tf_model->model->func_type) { + case DFT_PROCESS_FRAME: + //it only support 1 output if it's frame in & frame out + if (task->do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); + } else { + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + } } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; } - } else { - task->out_frame->width = outputs[0].width; - task->out_frame->height = outputs[0].height; - } - break; - case DFT_ANALYTICS_DETECT: - if (!tf_model->model->detect_post_proc) { - av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + break; + case DFT_ANALYTICS_DETECT: + if (!tf_model->model->detect_post_proc) { + av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + return DNN_ERROR; + } + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + break; + default: + tf_free_request(infer_request); + + av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); return DNN_ERROR; } - tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); - break; - default: + task->inference_done++; tf_free_request(infer_request); - - av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); - return DNN_ERROR; + av_freep(&outputs); + ff_safe_queue_push_back(tf_model->request_queue, request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } - task->inference_done++; - tf_free_request(infer_request); - av_freep(&outputs); - ff_safe_queue_push_back(tf_model->request_queue, request); - return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params)