From patchwork Fri Apr 30 03:07:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Fu, Ting" X-Patchwork-Id: 27534 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a11:4023:0:0:0:0 with SMTP id ky35csp2147261pxb; Thu, 29 Apr 2021 20:17:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzba+as5MaD8uFVTaPiL0ZBQO8Nt7DcMsbLKogJnO5NlJfEqfotqWdBp/lxB2EbUZfDPe1h X-Received: by 2002:a17:906:93f7:: with SMTP id yl23mr1724136ejb.407.1619752654695; Thu, 29 Apr 2021 20:17:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1619752654; cv=none; d=google.com; s=arc-20160816; b=Ch/dN0WPORVr3H/0xyJKbU4cLCLDv/aAsa3zNvROnYxw2VAxUIIAzB6uNN0yofIkwK YZbaYKBTK7tGcODla7IU+xL7llCg4XtsSVEpv8KIyJ8OUuxEw5EcSaCHKmI1pEyfmVIz JRapmnOB2HFGQ1bqszIRSNxPR5PFOiAzDbsB5MH9W1N/FrARLKt+udowwbTJh/vOzdLz Dftp+hbinbxnw72bQAaJxpewLFXXOhyejW29sDFFP4exloTBaH11S49xzjJaAaxSqHZ7 iHEzChapUeFEXOL1c29JdMtJvH11q9dZR+RhS250ijZm8TQkqd94RSgjly6u4qprh7Iw BejA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:ironport-sdr:ironport-sdr:delivered-to; bh=/zEcuBKcu5Lc0u+/AqRQJf35z8MVN55YvQpdbJU7FYg=; b=m3mU8WJbbsPiahfjvNYIkIwci9URCLyQvQ2DvTFUdhRi6hSRulSKQlcjMIQfHfADdW fTxkrfp5zTjONklO7vuqVWfVqQsDiZff02b9HtJ/ydJlzVfbO+OTcjurtabsGJZmCJkZ NP4DnSJ2fZ/NDXlbkJNpCyiGfs/EUv23NKkutlso3LchbpS4bfEHWVvIBbc7iLKoFpf8 NianvTRdvEWKMYeJJayXI1/s9ZMFzsfDUs3hP0kljL8/4VWuipRGf1fS/HhsvGuQQQku 8mjmtIF2fKCffNjPiWUo/3i6yeeBly5nIfBnJyPlWb45ob2vbxlWi4tyMwbFLBRTELIF c+cg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r26si1766880ejc.505.2021.04.29.20.17.34; Thu, 29 Apr 2021 20:17:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 37F3A68A2CC; Fri, 30 Apr 2021 06:17:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EDA3F68A2AE for ; Fri, 30 Apr 2021 06:17:07 +0300 (EEST) IronPort-SDR: ahyfgubMzNdjZDOI8PJuHHbO00a/BWFrzhQ1v5woMPZEcsGt1u/LlkadDl7NF8jUcz3W8MqZrY nM7f8g/OrFEg== X-IronPort-AV: E=McAfee;i="6200,9189,9969"; a="184659711" X-IronPort-AV: E=Sophos;i="5.82,260,1613462400"; d="scan'208";a="184659711" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Apr 2021 20:16:58 -0700 IronPort-SDR: vJUsd4HPIezZOfRuMQW3nFoA0EgCUw9gadASG9dBtsvEZ1E+OV91Typ0/MwMJ5ZL+qgThfkEhc OV7MsisQhBkQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,260,1613462400"; d="scan'208";a="459152508" Received: from semmer-ubuntu.sh.intel.com ([10.239.159.83]) by fmsmga002.fm.intel.com with ESMTP; 29 Apr 2021 20:16:57 -0700 From: Ting Fu To: ffmpeg-devel@ffmpeg.org Date: Fri, 30 Apr 2021 11:07:10 +0800 Message-Id: <20210430030711.30216-3-ting.fu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210430030711.30216-1-ting.fu@intel.com> References: <20210430030711.30216-1-ting.fu@intel.com> Subject: [FFmpeg-devel] [PATCH 3/4] lavfi/dnn_backend_tensorflow: support detect model X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: C9H1jykqTK3S Signed-off-by: Ting Fu --- libavfilter/dnn/dnn_backend_tf.c | 39 ++++++++++++++++++++++++++------ libavfilter/vf_dnn_detect.c | 32 +++++++++++++++++++++++++- 2 files changed, 63 insertions(+), 8 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 5c85b562c4..8fb2ae8583 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -793,15 +793,40 @@ static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_n outputs[i].data = TF_TensorData(output_tensors[i]); outputs[i].dt = TF_TensorType(output_tensors[i]); } - if (do_ioproc) { - if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(out_frame, outputs, tf_model->model->filter_ctx); + switch (model->func_type) { + case DFT_PROCESS_FRAME: + //it only support 1 output if it's frame in & frame out + if (do_ioproc) { + if (tf_model->model->frame_post_proc != NULL) { + tf_model->model->frame_post_proc(out_frame, outputs, tf_model->model->filter_ctx); + } else { + ff_proc_from_dnn_to_frame(out_frame, outputs, ctx); + } } else { - ff_proc_from_dnn_to_frame(out_frame, outputs, ctx); + out_frame->width = outputs[0].width; + out_frame->height = outputs[0].height; + } + break; + case DFT_ANALYTICS_DETECT: + if (!model->detect_post_proc) { + av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); + return DNN_ERROR; + } + model->detect_post_proc(out_frame, outputs, nb_output, model->filter_ctx); + break; + default: + for (uint32_t i = 0; i < nb_output; ++i) { + if (output_tensors[i]) { + TF_DeleteTensor(output_tensors[i]); + } } - } else { - out_frame->width = outputs[0].width; - out_frame->height = outputs[0].height; + TF_DeleteTensor(input_tensor); + av_freep(&output_tensors); + av_freep(&tf_outputs); + av_freep(&outputs); + + av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); + return DNN_ERROR; } for (uint32_t i = 0; i < nb_output; ++i) { diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 1dbe4f29a4..7d39acb653 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -203,10 +203,40 @@ static int read_detect_label_file(AVFilterContext *context) return 0; } +static int check_output_nb(DnnDetectContext *ctx, DNNBackendType backend_type, int output_nb) +{ + switch(backend_type) { + case DNN_TF: + if (output_nb != 4) { + av_log(ctx, AV_LOG_ERROR, "Only support tensorflow detect model with 4 outputs, \ + but get %d instead\n", output_nb); + return AVERROR(EINVAL); + } + return 0; + case DNN_OV: + if (output_nb != 1) { + av_log(ctx, AV_LOG_ERROR, "Dnn detect filter with openvino backend needs 1 output only, \ + but get %d instead\n", output_nb); + return AVERROR(EINVAL); + } + return 0; + default: + avpriv_report_missing_feature(ctx, "Dnn detect filter does not support current backend\n"); + return AVERROR(EINVAL); + } + return 0; +} + static av_cold int dnn_detect_init(AVFilterContext *context) { DnnDetectContext *ctx = context->priv; - int ret = ff_dnn_init(&ctx->dnnctx, DFT_ANALYTICS_DETECT, context); + DnnContext *dnn_ctx = &ctx->dnnctx; + int ret; + + ret = ff_dnn_init(&ctx->dnnctx, DFT_ANALYTICS_DETECT, context); + if (ret < 0) + return ret; + ret = check_output_nb(ctx, dnn_ctx->backend_type, dnn_ctx->nb_outputs); if (ret < 0) return ret; ff_dnn_set_detect_post_proc(&ctx->dnnctx, dnn_detect_post_proc);