From patchwork Wed Aug 25 10:39:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29774 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp435295iov; Wed, 25 Aug 2021 03:41:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxsFMXgt+kFxUrowcuMRUV+zcFbEIbWa9ryiLsk7Ymt4fqFxc8s+iJvadbbg6dgF2sttNHo X-Received: by 2002:a05:6402:d66:: with SMTP id ec38mr7355010edb.234.1629888087972; Wed, 25 Aug 2021 03:41:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629888087; cv=none; d=google.com; s=arc-20160816; b=Z5wUrCuAdLeO5FNjylKOw9McOUK5oD71dSHU95osWBdowWo9mj4iyy3mST6HJ7r5LA 3riwrVIpt0sKLWu+5ON9rsUdZw5DP9ez3Emc0pv9qmFsgOOmDf26g+nvPOaoLxbqOYA+ tFndMyt0XAYCCUtFl0eM1g1m9Ga28WUvuQJQB/jZkscpUKyDe/vS84bKxOqUrm6p0nNB JQPlt/5K+ddSXgNbVAL1tuXUB987JlHrVPJUsUCt5NSuemrPsoWKaCk5HAULk8bPkIrc 1swQLYjJ+PrwbibXq41bhjJBBqc3BWw/ZFU8/CKj3WLjNXJK2jQMk9yab6gLhhEczywN ilBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=nGY7fOqvCGAsNEZjXHpFF4Bs6DP2u7GhtsfYW3oYT64=; b=SsrUrfnIOxEqYzqSHUm5i7jzQjWX5kYyvI4lJ6cZpmSOMDviIwoK+5foeKdatdobaH Vs6y+9QAAHCLwUDlXxREGWU9AtzaHPG0fL/Gjq3Vvny7nR75/twvlqKkuCd4BCf0eQgC wHNhY9oBHN/LGhK+K8aPI80PnpeFRujG3rzDPvLCF2nWLdpVepkf456DC34NQzXl+nas vijVG/mu6ZIah5tWSaYfCOttfGHH2z5oWdAx9d4ljq0TQ03BVsUAXcs9GbZT0row6XWj pYlCd3u3YMxHJ/CoSkYmvAl+4yix+uqa4nL+/zbdhG/h3f+7fnwyNj/MV1c+3m/SjL4L 4VJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="f0TQ/Yv1"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w17si19358527ejv.123.2021.08.25.03.41.27; Wed, 25 Aug 2021 03:41:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="f0TQ/Yv1"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0FC0068A3B8; Wed, 25 Aug 2021 13:40:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5DF67680523 for ; Wed, 25 Aug 2021 13:40:24 +0300 (EEST) Received: by mail-pg1-f178.google.com with SMTP id c17so22630743pgc.0 for ; Wed, 25 Aug 2021 03:40:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=689fmjzc8ixjmruryRwS7yUk5dCh7YcqKpTxBlM4Xg4=; b=f0TQ/Yv1NJo6r8zD72JIZDpT6pxPHyq55FrEWXq6rvST7y6AtY2LwrtO+xlRjv3JzQ CeR+dOtCmzCvfqtxARmK3LI//5Y+o5aSvcJM3xnnClIRECXlCKGHLDcGoEz4HDnMlePl q6DDlYQtfaZLD3mH8iACq9XauKnYci3tfI/O6Wkj3g0AZGTgL0CiXWJIIARaLzjMJmPn lXvtVJ7HCku/a3EL0pC33Gw1VlYbUiLDzSfaOADro7OUfNcIbBw9wvxC6273ZMKd++ad 3m6rVvnwE6Bf/n9aB82goP1+5gLi4Lx57gW8yk8VqDNgeeBP/YoA/xzrq6o/6AP3pu8C av6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=689fmjzc8ixjmruryRwS7yUk5dCh7YcqKpTxBlM4Xg4=; b=hWVm1l/lr1ZWoNgmL7lbPorAb4a9httpL9kQcFqBmZGPkjOqjJ1vp/seAk6o1urZka poTMeMksvI9oH0sE1F+NuzT80R2ByY3pZCqWicHmM31QE1nhkrAL9fa1V83csYLTfSAo 1vRSpKN3YDzq5oOkDByX9gck5F9HzqYTEqY7mr/K7osqAQ+yjt2p0rHSnLhXG/7NUcsB Cer5/UIFhklrbo3j2WXa7gjEteE8IdKuyY8uVDwOnnkLvaU+6iPMWg6ZjizKXbM3vCd5 +z30FQ9DAxzeJnvdmyXYGNq2Hnd6DNZMNGvPWx9iZJ/r0/EbytqNP52Z6CzGyIN51d+V N7vQ== X-Gm-Message-State: AOAM531NPAtrRYnzGOcyTe3fbuy58YCHgvazzMsLF50KFSnc6GJAu44L e+PqtiQBbLoXnK/ZYDJBtaSDe09evEG7UA== X-Received: by 2002:a63:ed50:: with SMTP id m16mr41414469pgk.231.1629888022728; Wed, 25 Aug 2021 03:40:22 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.157.221.29]) by smtp.googlemail.com with ESMTPSA id w16sm4548735pff.130.2021.08.25.03.40.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Aug 2021 03:40:22 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 25 Aug 2021 16:09:40 +0530 Message-Id: <20210825103940.28798-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210825103940.28798-1-shubhanshu.e01@gmail.com> References: <20210825103940.28798-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 7/7] libavfilter: Send only input frame for DNN Detect and Classify X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: TdNhAVTfmI1p This commit updates the following two filters to send only the input frame and send NULL as output frame instead of input frame to the DNN backends. 1. vf_dnn_detect 2. vf_dnn_classify Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 2 +- libavfilter/dnn/dnn_backend_openvino.c | 5 +++-- libavfilter/dnn/dnn_backend_tf.c | 2 +- libavfilter/vf_dnn_classify.c | 14 ++++++-------- libavfilter/vf_dnn_detect.c | 14 ++++++-------- 5 files changed, 17 insertions(+), 20 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index d2bc016fef..6a9c4cc87f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -38,7 +38,7 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func return AVERROR(EINVAL); } - if (!exec_params->out_frame) { + if (!exec_params->out_frame && func_type == DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "out frame is NULL when execute model.\n"); return AVERROR(EINVAL); } diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 76dc06c6d7..f5b1454d21 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -272,14 +272,14 @@ static void infer_completion_callback(void *args) av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); return; } - ov_model->model->detect_post_proc(task->out_frame, &output, 1, ov_model->model->filter_ctx); + ov_model->model->detect_post_proc(task->in_frame, &output, 1, ov_model->model->filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: if (!ov_model->model->classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); return; } - ov_model->model->classify_post_proc(task->out_frame, &output, request->lltasks[i]->bbox_index, ov_model->model->filter_ctx); + ov_model->model->classify_post_proc(task->in_frame, &output, request->lltasks[i]->bbox_index, ov_model->model->filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -819,6 +819,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * if (model->func_type == DFT_ANALYTICS_CLASSIFY) { // Classification filter has not been completely // tested with the sync mode. So, do not support now. + avpriv_report_missing_feature(ctx, "classify for sync execution"); return DNN_ERROR; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index dfac58b357..c95cad7944 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -1069,7 +1069,7 @@ static void infer_completion_callback(void *args) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return; } - tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); + tf_model->model->detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; default: av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index d5ee65d403..d1ba8dffbc 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -225,13 +225,12 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); - if (out_frame) { - av_assert0(in_frame == out_frame); - ret = ff_filter_frame(outlink, out_frame); + if (async_state == DAST_SUCCESS) { + ret = ff_filter_frame(outlink, in_frame); if (ret < 0) return ret; if (out_pts) - *out_pts = out_frame->pts + pts; + *out_pts = in_frame->pts + pts; } av_usleep(5000); } while (async_state >= DAST_NOT_READY); @@ -258,7 +257,7 @@ static int dnn_classify_activate(AVFilterContext *filter_ctx) if (ret < 0) return ret; if (ret > 0) { - if (ff_dnn_execute_model_classification(&ctx->dnnctx, in, in, ctx->target) != DNN_SUCCESS) { + if (ff_dnn_execute_model_classification(&ctx->dnnctx, in, NULL, ctx->target) != DNN_SUCCESS) { return AVERROR(EIO); } } @@ -269,9 +268,8 @@ static int dnn_classify_activate(AVFilterContext *filter_ctx) AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); - if (out_frame) { - av_assert0(in_frame == out_frame); - ret = ff_filter_frame(outlink, out_frame); + if (async_state == DAST_SUCCESS) { + ret = ff_filter_frame(outlink, in_frame); if (ret < 0) return ret; got_frame = 1; diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 809d70b930..637874b2a1 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -368,13 +368,12 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t *o AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); - if (out_frame) { - av_assert0(in_frame == out_frame); - ret = ff_filter_frame(outlink, out_frame); + if (async_state == DAST_SUCCESS) { + ret = ff_filter_frame(outlink, in_frame); if (ret < 0) return ret; if (out_pts) - *out_pts = out_frame->pts + pts; + *out_pts = in_frame->pts + pts; } av_usleep(5000); } while (async_state >= DAST_NOT_READY); @@ -401,7 +400,7 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx) if (ret < 0) return ret; if (ret > 0) { - if (ff_dnn_execute_model(&ctx->dnnctx, in, in) != DNN_SUCCESS) { + if (ff_dnn_execute_model(&ctx->dnnctx, in, NULL) != DNN_SUCCESS) { return AVERROR(EIO); } } @@ -412,9 +411,8 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx) AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); - if (out_frame) { - av_assert0(in_frame == out_frame); - ret = ff_filter_frame(outlink, out_frame); + if (async_state == DAST_SUCCESS) { + ret = ff_filter_frame(outlink, in_frame); if (ret < 0) return ret; got_frame = 1;