From patchwork Sun Aug 8 10:55:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29346 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1980742ioh; Sun, 8 Aug 2021 03:56:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyq6PJf8HRvWGtx4OwJ88dP9eeQSXijQaU5YVy42/OigFH36Nk2vN4z5XqE8y/qMfiRc4gY X-Received: by 2002:a05:6402:3551:: with SMTP id f17mr8099443edd.261.1628420180172; Sun, 08 Aug 2021 03:56:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420180; cv=none; d=google.com; s=arc-20160816; b=mrKMoyZ2wI0EKEgPWUzNp+7NmZ9yGl/ifNjwH5/cUix1rJh/4x8GyPn7LRwnHYbv1R GJ9JALQAyZhUt3vbDAx3o3IGFsLDWinF5+VA68bsuZml6tOKLCRTgfoGIwdeuRM9xGjE Glq9QKrGwZNmB9S6mqje1v2hxHaFGZA+lyDhHVR224xEy3zMvAphZN7zmQb+Dt4r1Ov6 ltMXVmLjBuKL5C4Xk+e8T0JtnVSUF7lLVxf+64qi0X+whPvgwI8UymQsPgUM384VoeYU KYvajffGaTtUKIn4BAPDDFrHb/X4JoPieMQuG8rlbrvNG2Zy3vQf69HzQ2P9W0li0pzm uJ9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=XXNyPSZOvD+K8CoBc8u+zWJvCAk8Z5rN4jrX0qPogAKZacOJjUg3hgvlNvYCE4UZ2E O76IJslStpLGmXmHVU0pzLPCBwzz5Rp170GH4pPSnFhQ5/OV4csy08bbNXG4CzIiIfnq T/mrl0LVEz52t7JUcuCM4S8ZNx82ay+sVCiOJ3N0cjDyCGKQ3ggYQrqETJODuo99/roU ZnHh6WO93Jh6oeGj/LW4pRHsRnlmN1j5Jspdyw5SByU1ed4LRKjqw4qXqEmLby3mZKah MSHuUQHtRn459ibXZBloQghGVIwPTciQjbbW2hjAjvalaf/51PGL7jlHOVb52iEXsWBM Fs9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YCGI24oS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l15si10622566eje.462.2021.08.08.03.56.19; Sun, 08 Aug 2021 03:56:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YCGI24oS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 68B1D68A404; Sun, 8 Aug 2021 13:56:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A1E0768A000 for ; Sun, 8 Aug 2021 13:56:09 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id cp15-20020a17090afb8fb029017891959dcbso1989034pjb.2 for ; Sun, 08 Aug 2021 03:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=YCGI24oSj/b+7Yx33gjdW6LGhHPohlPl227lJeeWlw5yLiYibg/iZG7SSA27FEY4ZS dstJkLoSvLsPvxwvgGcwrwaUhiXfGA9ZOW07MDmYggV/M2Y9puKOY8Xulb6aO8S6Tc9W pcDPfdPdSofpx21ItUViK0ACTsRMNfrt5GCT4WjgIisg2I2ccEsN1VyfQXlHmnDRDMuS q1fyiRN6FKZSSIQB9LzEa7Mom+l9PFlYIsZZ+yD8yTNvOZDR1jlkNdpWuuqX4aJrB+E5 ++IYM9sy2wyDjK8VCu2SubJNVPRH4Jp3e/BDWE6ooQzpzFjnIHZ94Wgr6VkkqQ6K2mmT kQAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=OUBoVrQDoVfSgl0cU01btBp4keAMyPhLh5lmWM6f01glLQ2fqCAP2LFxB6rQUk5YNu G0ER9MthuWZnsof1IQDSBlYmuIcZ14ZV9Eabp2nZB2Sz4nsn7Fr9S4VpA95Qufz7ElrE hgQcciu7rp/dD5rzBBFRQPeLX0nQtmvcc3bjZ3/u/byTujscEAuGBa4brpQU381kZLy+ nkDe5sP0D6BDc2sBw6eF6+bEykQ+kaCFuZ2e3xpEea3wyhxhFMMG3AA7EJuChCmPn3O2 fE2QwobZyQXmh4X2wc8i3gQFqfUBjTDydPA/9iCkC7KQCVKmNi34oZGoQkkacDIGXcx9 l6tw== X-Gm-Message-State: AOAM533GF4aFGo+frzafPIcMQse77pZvmQ6OpyKmmX7hBDw7cFaByXRO mF8IU3xt/jaB6s8TXYE5yJ0rK8sNwB7eaQ== X-Received: by 2002:a17:902:8f8b:b029:12c:d085:7918 with SMTP id z11-20020a1709028f8bb029012cd0857918mr16666866plo.10.1628420167564; Sun, 08 Aug 2021 03:56:07 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:07 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:31 +0530 Message-Id: <20210808105539.18822-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/9] [GSoC] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +SJr1ZdRaUNW This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif From patchwork Sun Aug 8 10:55:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29344 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1980820ioh; Sun, 8 Aug 2021 03:56:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzMMHoasJcJwTKdpceIVvHnVi2v7jFT4g6Oxs3lddO0i98/d315CDuYh9HkkV/NEQ+thbHI X-Received: by 2002:a05:6402:361:: with SMTP id s1mr23082465edw.172.1628420190085; Sun, 08 Aug 2021 03:56:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420190; cv=none; d=google.com; s=arc-20160816; b=ZK8tyWG/9djOyEgGkn31e1N/fgW/AQA2eEmfV7ISIx/dfxmBEdI7xrzk/GQ87pAvNp ay63IrlxWxn/+4RuOY9NpNMT19to/NytcAlFn0QO65t9u9MvTPbnJn65OIt0uAVbqTZk mq3nFb0g3maW7YslA4CVrI0KSPSKF3qWQZyuitITHQ2pKV0OpG5KiadNArou07jZ2i9r X0dRxah+qoPQxsASwTKxDk7q7Q0IWH1KVzFt9Ci7WXiabp/bChMEwd7NeJgMBRgPU3vV JRjVVcR5imzQrshz3zlgxyS4oARRyafnpVRjVoWueBlioDeC8U7t16LZiPRa+rHPmyJC i4SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=2BaZt/nZ8tYkB2dMxu8/Sj+l4fN6MVIaca0p0tqv2lg=; b=tDowgRi/rsjnyGTa1YntKOqeVhfxrJY8sFOq66S2Tuv3wiCC1/h0MCXZhsPUPb4wuR knOGHWEUXFv7Kx+MSX5ga8Dt57QGYZiWZeXu9G7rbIueznP4D2501zwez0Bj4ti6AeEQ idKshiTfpSIzaPmTI3rlWYPppVz0py3R0r5gnljE1l4gMJpWTXei1Qrr96qIEzaII34V H0kG1+RCe750AfqeYhuVdMieJEdqTrQ1BoTMmOORkPQ9uXhggIQ4Ho1OtY6M8g6N6DSs I5CCdA6gOXrmOoK59wF9aZ+0wvGbK0/Q/oaIgd5G37imm6ormK4vgkxbOaGZ5mvAazOC sh/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OYTWeduB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id o13si10950201ejh.129.2021.08.08.03.56.29; Sun, 08 Aug 2021 03:56:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OYTWeduB; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A0CA768A451; Sun, 8 Aug 2021 13:56:17 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4225C68A000 for ; Sun, 8 Aug 2021 13:56:11 +0300 (EEST) Received: by mail-pj1-f52.google.com with SMTP id u21-20020a17090a8915b02901782c36f543so20916743pjn.4 for ; Sun, 08 Aug 2021 03:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=OYTWeduBk9k3tE6+6ebR5cvSUdwSCWFwD8yMoNTPGoqa1yEj6eAAymaraDyfNbg9Wn KDIxtzXrUsFMJnYUVEHa0iIAoulN9x4G+z4+R9otwJmGgqlwXIFi0JAQwneDk6jJyudI vZAhHZMb86Iwdr0ZUzdIIBAiW8vR5dNaEm9q12lERSMX/X5obMKyP0q2kmci++K4xKmD stuxuwgeEzCvm30yIyYS8AGZn+T23W9mMlQYPOl0hupp5tn5y2VvYvTFnmb1E3/Jz2Di RxZHQ1nP3Cps8S+WAvrxioDPKdpBQyB78K0jdEWGsxIu1nIID++Er2NDqJt5mXGhJHSy WLCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=rvWMHbcuu2TFCwuPLkZWQ9Ed2/H0D1jxUmcJyZOKqLvz1AGlAgHh1LfDkyJub2VPg/ F7cea/fWHYTcBKlYQLuNGCNW2QDAEC9GIHfFbvb9D6Ehp9rM8RQT+12qh7hFCOatiBPI a8MdZD27iI4X1eabx1DFCokiBgMpFkEoX6xjXMITShYZM2rIio4MdZkQYsRfjDxAEjun Z/4FhicQrbEg+GRt8GejQApz0zGfZUcN8LGMwqUiaVlmwcs23Kvw0m0Kpf543853/Htj D8wr1Pornbm+QUWmsAqUm0LjesbYEPG518yqoWYyVpiOQFgHL2LxAfFjDKIpzClryFu8 Y9Fg== X-Gm-Message-State: AOAM531YHmTVZskrmDdAK9ppGxutOaimjyw71ZkTBuKP384x5/c9iXK/ S5g0LEhuMbJbqQuT4KZR6E9Tg01MyMaatA== X-Received: by 2002:a17:90a:eb0c:: with SMTP id j12mr30116320pjz.116.1628420169309; Sun, 08 Aug 2021 03:56:09 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:09 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:32 +0530 Message-Id: <20210808105539.18822-2-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 2/9] [GSoC] lavfi/dnn: Common Function to Get Async Result in DNN Backends X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Qo1urrMVBSiF This commits refactors the get async result function for common use in all three backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 20 ++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++ libavfilter/dnn/dnn_backend_openvino.c | 18 +----------------- libavfilter/dnn/queue.h | 1 + 4 files changed, 37 insertions(+), 17 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4c9045501f..df4bab85e0 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -122,3 +122,23 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ #endif return DNN_SUCCESS; } + +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out) +{ + TaskItem *task = ff_queue_peek_front(task_queue); + + if (!task) { + return DAST_EMPTY_QUEUE; + } + + if (task->inference_done != task->inference_todo) { + return DAST_NOT_READY; + } + + *in = task->in_frame; + *out = task->out_frame; + ff_queue_pop_front(task_queue); + av_freep(&task); + + return DAST_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 96e4df9676..e1d678b230 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -24,6 +24,7 @@ #ifndef AVFILTER_DNN_DNN_BACKEND_COMMON_H #define AVFILTER_DNN_DNN_BACKEND_COMMON_H +#include "queue.h" #include "../dnn_interface.h" #include "libavutil/thread.h" @@ -122,4 +123,18 @@ DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); */ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); +/** + * Extract input and output frame from the Task Queue after + * asynchronous inference. + * + * @param task_queue pointer to the task queue of the backend + * @param in double pointer to the input frame + * @param out double pointer to the output frame + * + * @retval DAST_EMPTY_QUEUE if task queue is empty + * @retval DAST_NOT_READY if inference not completed yet. + * @retval DAST_SUCCESS if result successfully extracted + */ +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 8d23446296..47cbd48ba2 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -32,7 +32,6 @@ #include "libavutil/avstring.h" #include "libavutil/detection_bbox.h" #include "../internal.h" -#include "queue.h" #include "safe_queue.h" #include #include "dnn_backend_common.h" @@ -883,22 +882,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa DNNAsyncStatusType ff_dnn_get_async_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) { OVModel *ov_model = model->model; - TaskItem *task = ff_queue_peek_front(ov_model->task_queue); - - if (!task) { - return DAST_EMPTY_QUEUE; - } - - if (task->inference_done != task->inference_todo) { - return DAST_NOT_READY; - } - - *in = task->in_frame; - *out = task->out_frame; - ff_queue_pop_front(ov_model->task_queue); - av_freep(&task); - - return DAST_SUCCESS; + return ff_dnn_get_async_result_common(ov_model->task_queue, in, out); } DNNReturnType ff_dnn_flush_ov(const DNNModel *model) diff --git a/libavfilter/dnn/queue.h b/libavfilter/dnn/queue.h index 2524d5fa59..9ab5c89f65 100644 --- a/libavfilter/dnn/queue.h +++ b/libavfilter/dnn/queue.h @@ -18,6 +18,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #ifndef AVFILTER_DNN_QUEUE_H #define AVFILTER_DNN_QUEUE_H From patchwork Sun Aug 8 10:55:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29349 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1980913ioh; Sun, 8 Aug 2021 03:56:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzU8Ig27UConUtiLrqpnGJiPTi1dndqsm1bx1DL/RrVoJiBYKMZc+NycTPFlRNPBnIPiYG5 X-Received: by 2002:a05:6402:184b:: with SMTP id v11mr23581950edy.267.1628420198747; Sun, 08 Aug 2021 03:56:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420198; cv=none; d=google.com; s=arc-20160816; b=BKAFsRSLlRoWWBNcg9HBhsqkzjk0IVBEQvHD3mpqn9w4vS/KkfPcMLbDBZ/4ay8qhq V2jsdtiorOZbbS1ZrAd48tPnBzjLDI8LMUBUmbwSSDCqVxgBI4HJu1MmuTtZqVsz624D SsLsY3ybx9K7vijOGTH5462q1vgzLoKvSRvsXVMdO1WIApP2KWRcSTOTzJGXgbmDlK7c mw0gFD7q4r7mrTBIpPiqaO76VMpYlx7FMeX+A931z9S5DOP3OOBhlOkRSCrxZNVaz6dr 9wBb7SvNOhb4qL6NZvFvulE9SIf1EqNKM9LzOXKW6B2c1cCg+PDqJWjxx9kKxadTcD8w 3Ulw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=YMUTaUfel1J7hb5Ymw4unP1mrHBl8tlnblqLWs6WETQ=; b=B8l93QioJ7+QYHvdugPg64qEDcvjTmhIVT4fpW9DJBllshuND58kNIcqRDS1uZIgWE aoUGkS8kiuW9sfw5+Zc3UFKElXk/I7NJ86s8FDGEqiRk1ljH/3DBXA8Uhxn99M0/BcOW sPw0EBuSrhqUE9iCFjCMoG9LXWsjvJv/QHu4sCUDmiXLTTj78FKul9T2DvfSq+fBjOPV a7gV+KCVCyVdYigAJner+49ymWnqKENZ5aqjSQFtEWzzosRFwF8nmQlb6TFUjeOp/pSP 1Z3VvzB8vE8sZv/enSOOsI3eLY/xVSJLjCDqpnKGivr+XL6PAeCxLTa03xUfANibFj4S MfOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=hyNrRVMT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id f14si14103833ejx.618.2021.08.08.03.56.38; Sun, 08 Aug 2021 03:56:38 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=hyNrRVMT; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C24FF68A466; Sun, 8 Aug 2021 13:56:19 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C1D7968A457 for ; Sun, 8 Aug 2021 13:56:12 +0300 (EEST) Received: by mail-pj1-f51.google.com with SMTP id mt6so23460593pjb.1 for ; Sun, 08 Aug 2021 03:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=coEvR7NJ3/O2PciedLn3eoYxS1n8ONZdZczMZWtj/AQ=; b=hyNrRVMTlteS8BQm30ovMJJRh11ucBFIZQLOESisZV+B31f7i34AHDSAKUBQdYPk/G jyd3F2SksvdQsSsuDD5KhRXf5zkMeFh/i+aAK1aXGvssQhEyRkcX9qvMpwLyQvnHi4P7 KGP+A2e9JzdW8n04OwMc9nvyQWV03LvwFFmiE2zgCssxq2QUwwjrhJxUrUhuVdw4HxDJ CQnEzoCz+N543ynK2aj5Zmjiyl77EEwWvNNPOs8PSO6UlhGNgMwC8r2Kpwn8MB/K/IZm g6EN83563qHwDR/nLDTuUsf8vRRL/TQxc9q6+KhATD4A0wFK0E3GfJLL9CUNWL25wNR/ 8Scg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=coEvR7NJ3/O2PciedLn3eoYxS1n8ONZdZczMZWtj/AQ=; b=HxFeMBm2o+12Mbru7JjkPsCJZyj2OJmcjOuGycMkU5+R/lD7DG77ckZuY0IO1UhEWX koKRPPNKE8F1Qi3oHmvMz0j3CWeT2cBCZ0z58mVBsp/O8U64MKLIAepDT4i4b3AHUUGy UkubG+mk80m9ZQfuM/FandS2q27+GrXKojWNyNJwA6VKBw7voqn6QVnXMyseAKF6CkRp C9pWAp4cd4LJUDXGM1+tGjMGNX9XDsPu+oyujaF9nvs2nTXxMMDVKHqM/F9yWBirf98v AhXUxmyUK5tmzaTOU/vmzsWkzqBWB5g32c45Mx7tOdCXGlyk3PEBc+VzgfPtKkDsAOSq qg1Q== X-Gm-Message-State: AOAM53062nfKlaCm0NC44aQUjIwqc5b8uLQzGG0mzh6kAkNnBCUe7fKI k+WkwC9PICnp7n8D7s/3dO4F9gWlzysGIw== X-Received: by 2002:a17:902:8642:b029:12c:dda2:30c3 with SMTP id y2-20020a1709028642b029012cdda230c3mr8989635plt.84.1628420171099; Sun, 08 Aug 2021 03:56:11 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:10 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:33 +0530 Message-Id: <20210808105539.18822-3-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 3/9] [GSoC] lavfi/dnn_backend_tf: TFInferRequest Execution and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: UuermNQl0qKl This commit adds a function for execution of TFInferRequest and documentation for functions related to TFInferRequest. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index ac609ab9cb..939afec41d 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -95,6 +95,13 @@ static void free_buffer(void *data, size_t length) av_freep(&data); } +/** + * Free the contents of TensorFlow inference request. + * It does not free the TFInferRequest instance. + * + * @param request pointer to TFInferRequest instance. + * NULL pointer is allowed. + */ static void tf_free_request(TFInferRequest *request) { if (!request) @@ -117,6 +124,12 @@ static void tf_free_request(TFInferRequest *request) } } +/** + * Create a TensorFlow inference request. All properties + * are initially unallocated and set as NULL. + * + * @return pointer to the allocated TFInferRequest instance. + */ static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); @@ -127,6 +140,38 @@ static TFInferRequest *tf_create_inference_request(void) return infer_request; } +/** + * Start synchronous inference for the TensorFlow model. + * + * @param request pointer to the TFRequestItem for inference + * @retval DNN_SUCCESS if execution is successful + * @retval DNN_ERROR if execution fails + */ +static DNNReturnType tf_start_inference(void *args) +{ + TFRequestItem *request = args; + TFInferRequest *infer_request = request->infer_request; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + TFModel *tf_model = task->model; + + if (!request) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); + return DNN_ERROR; + } + + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + return DNN_ERROR; + } + return DNN_SUCCESS; +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; From patchwork Sun Aug 8 10:55:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29347 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1980992ioh; Sun, 8 Aug 2021 03:56:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwO0SafHm1QUNuaUmFF2vEE2eSdImg8Knzmf0toC55SviKr2fLaooXt//sgQclfWyylWBlI X-Received: by 2002:a05:6402:35d2:: with SMTP id z18mr23854351edc.282.1628420207787; Sun, 08 Aug 2021 03:56:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420207; cv=none; d=google.com; s=arc-20160816; b=eXMq3XcoGakWfnAYyHob49is4FJGebLjWSBylJ+45GAS/jQxAKU+Fgbpk8XK17Iq9Q OFHRQNZ5hiVXYt8tYnKRaYoT0i9DRNqZ0WSLp7Yqzx5ynXmfDN8SNNCcfHfvhjw0RSMo vynRUKWatJcr+DglTqiBWeEjGoyPxW4OyagLzlFBx+3BHL6JDZRn3X/4QjojP/Z5uZN7 TxQCjk4+C4Q7mGs0CAT6cI8HkbdEg7Wyd4fYZ6UTpTjF9wgLTa6EUuTJ+nxYn7byglxe V79qc0dbSxVvLQQuvrwLyFmYNiHpIIGmPBIKWf4g4P3aNiBM8nIfihG3/Gw9Ks0TqRll GrHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=EQ5Rp4Ke1v+pBESbcAlA62yKC/8iyyCbVrI1wFUUVuk=; b=Bv/EhOqG3ygrbqboD2+VRpzxjDH2fGDJ5MsV0/qvnu+XzV+r2b8Bmei8JN0OdZK5ea gNxrAwEpxLWvgd2Su0qNRGwpXm2DfvahUxEtp3VELppyQ2UUxpsZ0/Em9rHun+Fdlh7e is3RSYzNSFsI+r7rif39Ewzzx2/iMrJGsYne4lKRmFxPM8gQijxQHgGwUiScRU9ZhbyJ d3VGGUdjfUXk49ROEizonZtMzfFeXxni8gUEy4yQHdYjwQYFAaIXnGeikeMJY9PcQATv kbILRcazkxM1Vi2ZBi7g5ztxNqxE6dJt8gVgvKU/vYNvZZsyCWwcqIuXhxkVcQTTUeV+ P/+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=nycCB+gJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x10si13455858ejs.36.2021.08.08.03.56.47; Sun, 08 Aug 2021 03:56:47 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=nycCB+gJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DC21868A484; Sun, 8 Aug 2021 13:56:20 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9F65F68A12D for ; Sun, 8 Aug 2021 13:56:14 +0300 (EEST) Received: by mail-pj1-f41.google.com with SMTP id ca5so23419039pjb.5 for ; Sun, 08 Aug 2021 03:56:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N3QNNHB0Se/VWJAk/EyGjhBwmSBSBktjvDjtHKv+6vs=; b=nycCB+gJlW7PAjJ5m4fOrBfvmzC0ivklUlV+tB07BMrAeEFREGMDwyqJ151kjYC85n x/62Ogq/KQaaSPE4dUxWbWoLPxiTvp6Zs0hkocj011HVeMMWsY1i0mt2VM6pRv4evO0m YCD2udcPT+Vkf38lZif9OTA37g++UiFIOspxpquUFhBuhNBTOqc/jEQ5usBie73oBSFv wk/B/aqMTInhP7BrTLEiMa967W6Qv5fZTw5cyOD4bQ1EmB6HWsBreY0awSWgmQ6sbTv4 OdSfbmyTpwJxPu4yDehxQI+8G/2la6Gz8fTGQg/OITpuTEi4e8AzpiLOA8GK9kEJ3YBB avRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N3QNNHB0Se/VWJAk/EyGjhBwmSBSBktjvDjtHKv+6vs=; b=JmMTBFZHD7TO5fz05eygxXKSh3jsS5HFsV41crCNVX+5ZZMzLIZ1/kYlz8uSHk2zu0 1KofUhOnMkD3QLmiJwrF/ktOx2Cv8AClCcanWyZLc3LG2Q/+OWNsO6vCUQI77YjnPkC2 hMZg4K2C/GkBx5jf1/huO+7r6Z3psKaxd5Mk6GNd1dwBRN6QTMiQgaINmLMOT5w2lHV9 VT3opDmFuBPnp+ZNB23Te4PSHYoUOdMKBM6l462w1RsyNQOlxWWcZUTeAv4uz1TiOKnV cvli6Z5If9iVIvFEjb77RXz+Tu6MWk35NHZFTf2ZFfSscrDEMeR+CCrgJ02ACmDgRAJZ tedw== X-Gm-Message-State: AOAM530FcVwJkUh2jg5LnnTdPUyQQwsji/xoxrFFNcqthpdGA78F7I8/ 5RQL2Ygfxydjf4sagpUHNeCkGrSymsnbCA== X-Received: by 2002:a17:902:c20c:b029:12c:ef04:faa3 with SMTP id 12-20020a170902c20cb029012cef04faa3mr10686625pll.44.1628420172832; Sun, 08 Aug 2021 03:56:12 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:12 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:34 +0530 Message-Id: <20210808105539.18822-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 4/9] [GSoC] lavfi/dnn: Async Support for TensorFlow Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Wr0zlPrE9a5N This commit enables async execution in the TensorFlow backend and adds function to flush extra frames. The async execution mechanism executes the TFInferRequests on a separate thread which is joined before the next execution of same TFRequestItem/while freeing the model. The following is the comparison of this mechanism with the existing sync mechanism on TensorFlow C API 2.5 CPU variant. Async Mode: 4m32.846s Sync Mode: 5m17.582s The above was performed on super resolution filter using SRCNN model. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 121 ++++++++++++++++++++++++++----- libavfilter/dnn/dnn_backend_tf.h | 3 + libavfilter/dnn/dnn_interface.c | 3 + 3 files changed, 109 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 939afec41d..3089ce8194 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -38,7 +38,6 @@ #include "dnn_io_proc.h" #include "dnn_backend_common.h" #include "safe_queue.h" -#include "queue.h" #include typedef struct TFOptions{ @@ -59,6 +58,7 @@ typedef struct TFModel{ TF_Status *status; SafeQueue *request_queue; Queue *inference_queue; + Queue *task_queue; } TFModel; /** @@ -75,7 +75,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; - // further properties will be added later for async + DNNAsyncExecModule exec_module; } TFRequestItem; #define OFFSET(x) offsetof(TFContext, x) @@ -89,6 +89,7 @@ static const AVOption dnn_tensorflow_options[] = { AVFILTER_DEFINE_CLASS(dnn_tensorflow); static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue); +static void infer_completion_callback(void *args); static void free_buffer(void *data, size_t length) { @@ -886,6 +887,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->exec_module.start_inference = &tf_start_inference; + item->exec_module.callback = &infer_completion_callback; + item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { av_freep(&item->infer_request); @@ -899,6 +903,11 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ goto err; } + tf_model->task_queue = ff_queue_create(); + if (!tf_model->task_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -1061,7 +1070,6 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q { TFModel *tf_model; TFContext *ctx; - TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; @@ -1074,23 +1082,14 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q tf_model = task->model; ctx = &tf_model->ctx; - if (task->async) { - avpriv_report_missing_feature(ctx, "Async execution not supported"); + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { return DNN_ERROR; - } else { - if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; - } + } - infer_request = request->infer_request; - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + if (task->async) { + return ff_dnn_start_inference_async(ctx, &request->exec_module); + } else { + if (tf_start_inference(request) != DNN_SUCCESS) { return DNN_ERROR; } infer_completion_callback(request); @@ -1127,6 +1126,83 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * return execute_model_tf(request, tf_model->inference_queue); } +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TaskItem *task; + TFRequestItem *request; + + if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { + return DNN_ERROR; + } + + task = av_malloc(sizeof(*task)); + if (!task) { + av_log(ctx, AV_LOG_ERROR, "unable to alloc memory for task item.\n"); + return DNN_ERROR; + } + + if (ff_dnn_fill_task(task, exec_params, tf_model, 1, 1) != DNN_SUCCESS) { + av_freep(&task); + return DNN_ERROR; + } + + if (ff_queue_push_back(tf_model->task_queue, task) < 0) { + av_freep(&task); + av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); + return DNN_ERROR; + } + + if (extract_inference_from_task(task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + return execute_model_tf(request, tf_model->inference_queue); +} + +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) +{ + TFModel *tf_model = model->model; + return ff_dnn_get_async_result_common(tf_model->task_queue, in, out); +} + +DNNReturnType ff_dnn_flush_tf(const DNNModel *model) +{ + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TFRequestItem *request; + DNNReturnType ret; + + if (ff_queue_size(tf_model->inference_queue) == 0) { + // no pending task need to flush + return DNN_SUCCESS; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = fill_model_input_tf(tf_model, request); + if (ret != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_freep(&request->infer_request); + av_freep(&request); + } + return ret; + } + + return ff_dnn_start_inference_async(ctx, &request->exec_module); +} + void ff_dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; @@ -1135,6 +1211,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + ff_dnn_async_module_cleanup(&item->exec_module); tf_free_request(item->infer_request); av_freep(&item->infer_request); av_freep(&item); @@ -1147,6 +1224,14 @@ void ff_dnn_free_model_tf(DNNModel **model) } ff_queue_destroy(tf_model->inference_queue); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } diff --git a/libavfilter/dnn/dnn_backend_tf.h b/libavfilter/dnn/dnn_backend_tf.h index 3dfd6e4280..aec0fc2011 100644 --- a/libavfilter/dnn/dnn_backend_tf.h +++ b/libavfilter/dnn/dnn_backend_tf.h @@ -32,6 +32,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out); +DNNReturnType ff_dnn_flush_tf(const DNNModel *model); void ff_dnn_free_model_tf(DNNModel **model); diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 02e532fc1b..81af934dd5 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -48,6 +48,9 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #if (CONFIG_LIBTENSORFLOW == 1) dnn_module->load_model = &ff_dnn_load_model_tf; dnn_module->execute_model = &ff_dnn_execute_model_tf; + dnn_module->execute_model_async = &ff_dnn_execute_model_async_tf; + dnn_module->get_async_result = &ff_dnn_get_async_result_tf; + dnn_module->flush = &ff_dnn_flush_tf; dnn_module->free_model = &ff_dnn_free_model_tf; #else av_freep(&dnn_module); From patchwork Sun Aug 8 10:55:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29343 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981082ioh; Sun, 8 Aug 2021 03:56:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyaUBB4p/sYo3pxTqClo4qbcQb+HmCsFYOP8xVZTdinktsU0iw0UWWAW6CgRFsIaO2g+LGb X-Received: by 2002:aa7:db94:: with SMTP id u20mr23238845edt.381.1628420217677; Sun, 08 Aug 2021 03:56:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420217; cv=none; d=google.com; s=arc-20160816; b=qXgLbQ7/YNHgwmQFYYdmCNohv8fkfkAjA13SdYR0krqOiROeVtZ9gsryTi69Ukz1XW 4bSAOtW9Qro11orfbFlB5yfWiPlvsNJQagtG2e4QJfsggLjNKetCx/YxsrY6e/KnSqNK OHbOU/3sHyzTJAIZETx2g5Qvk1Noz4+rULVsjBB8focGpo0p+4V4AyDTCkKhGulGsBqC EOYf18NdfDMdqMPUVgIX7NajvHlW7HytQO8yxCKGf4aIkC+LFBGglUdL+yBoANyzOT2V pkJL9efZ59WEoUD5RYnEkUSrMDkCCA0rtAZ5xve3QKmaS/GPRZbHsC3p3GNN5p6U8DnV ZmrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=3ZPNhoSmik2i9s6Hg1XxwOOl9Ef1DuxnNNC+h/FzzP8=; b=OzkvdcRN4Md/LRhpR7ewbxFmt/usBdOYP36Q6wBBjRoLdHkdHySaDw3UYpu1ksK2aC v5TGbjDuVO8+sKoLW8wTofB1IfVR6H0bQ9KfEocfdfQynflouPMr0rB9QGhdysBByhY8 5bwuKkb1LrOKosKCT5cfEiOQOvsZtFfPoLBzJDRkEPz+CgCjog72ETzFNT5xphEyHfYX JEyvN7jJgedMC5soDZYLx+3TLleEOA2dG0f6Fbd/g9UHtZff3d8GoZGQeH0ZHV7Y+bWQ mM0HHNsi71cQ+VBcWiOsRpuckVTiRF8AQVnO0Lq7eLnX1/koMs0oe9DlITOsfMRUYrAI vGDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aUvdyWcc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s13si14660170edd.363.2021.08.08.03.56.57; Sun, 08 Aug 2021 03:56:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aUvdyWcc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id EFB9868A4A8; Sun, 8 Aug 2021 13:56:22 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 93F0668A49E for ; Sun, 8 Aug 2021 13:56:16 +0300 (EEST) Received: by mail-pl1-f177.google.com with SMTP id c16so13338311plh.7 for ; Sun, 08 Aug 2021 03:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OLxfPF/rWh6p6YqLopR9KSquSYZfnYtXySIFUvPcXqA=; b=aUvdyWcc4m6zvJTN4GA+AOkGcS52PdmaQ0snuU2aKPB3oYXurG5ztavFFTwbhPbTz+ CRUsgOzkWaCoVBnaits8OrtxLlLtneVlZqtkLF+BDAS2umohbAOkxb4O1TNogXi9y8eO kfNbBRBeTeOy0pEGtcj4oh575yCedFMeCdxLBuuSchtYGIR7bxIrr7O9Ks7fICYlPn/s +LkrkVkJUfPtF7unl4dXXNHuADoogUb56bfhMocG3GBS6NThzeY66bC3HLeIgllQWH1G bJadZYPyW7fijtN+hS40+pZ6rzZHCjlP/aYQ+QwHodbEg8+hkuK1wQwqIJMYb7sxCwEi EY6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OLxfPF/rWh6p6YqLopR9KSquSYZfnYtXySIFUvPcXqA=; b=C/lfl0ttbQFAEQAcU7/B/PdybMdkTl8FW7Zfrrq7tw21fd0yMcwVumoMWLYclFlfUr vjTuztqwEChlIYbBzUmQkQpCzmUMQ2AYN2Fbzz46jUaq3qiLfeABx9kFvv1nKpRvdVni Njdp7CcQMNHVZUJenCt+jPORj2geJFCELf+cvUtee84t1bBwLMIbEXIn+6YQPswNax4g 9W+ex0OLLioXM9E6JfVaWaAs35UCcaNXMCpszaW1Hc0PN+w+v2dYkUwDofsDeEjVZ8IM 24VqC0qTuAW7NrZjN2LRnHGGwxRo0ny9Qh2lgVVlvO68NQUTwQR6l6hsz+sWXW8NkV+K EA7w== X-Gm-Message-State: AOAM532f+XpGsKrUrcRvvzR12iWz1ZoE9O1a6mFoOZOI2w+FNTpIsNdO VtzubYpRkC4mR0hHGxtnSASVtTjLaYYAGw== X-Received: by 2002:a65:4008:: with SMTP id f8mr173551pgp.310.1628420174567; Sun, 08 Aug 2021 03:56:14 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:14 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:35 +0530 Message-Id: <20210808105539.18822-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 5/9] [GSoC] lavfi/dnn_backend_tf: Error Handling for execute_model_tf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 2apaGwo7BcZj This patch adds error handling for cases where the execute_model_tf fails, clears the used memory in the TFRequestItem and finally pushes it back to the request queue. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 52 ++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 3089ce8194..ad5f1e633e 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -173,6 +173,24 @@ static DNNReturnType tf_start_inference(void *args) return DNN_SUCCESS; } +/** + * Free the TFRequestItem completely. + * + * @param arg Address of the TFInferRequest instance. + */ +static inline void destroy_request_item(TFRequestItem **arg) { + TFRequestItem *request; + if (!arg) { + return; + } + request = *arg; + tf_free_request(request->infer_request); + av_freep(&request->infer_request); + av_freep(&request->inference); + ff_dnn_async_module_cleanup(&request->exec_module); + av_freep(arg); +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; @@ -881,6 +899,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ if (!item) { goto err; } + item->inference = NULL; item->infer_request = tf_create_inference_request(); if (!item->infer_request) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for TensorFlow inference request\n"); @@ -892,8 +911,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); goto err; } } @@ -1060,8 +1078,7 @@ err: av_freep(&outputs); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); } } @@ -1073,28 +1090,35 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q InferenceItem *inference; TaskItem *task; - inference = ff_queue_peek_front(inference_queue); - if (!inference) { - av_log(NULL, AV_LOG_ERROR, "Failed to get inference item\n"); - return DNN_ERROR; + if (ff_queue_size(inference_queue) == 0) { + destroy_request_item(&request); + return DNN_SUCCESS; } + + inference = ff_queue_peek_front(inference_queue); task = inference->task; tf_model = task->model; ctx = &tf_model->ctx; if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } if (task->async) { return ff_dnn_start_inference_async(ctx, &request->exec_module); } else { if (tf_start_inference(request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } infer_completion_callback(request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } +err: + tf_free_request(request->infer_request); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + destroy_request_item(&request); + } + return DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) @@ -1194,8 +1218,7 @@ DNNReturnType ff_dnn_flush_tf(const DNNModel *model) if (ret != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); } return ret; } @@ -1211,10 +1234,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); - ff_dnn_async_module_cleanup(&item->exec_module); - tf_free_request(item->infer_request); - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); } ff_safe_queue_destroy(tf_model->request_queue); From patchwork Sun Aug 8 10:55:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29350 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981183ioh; Sun, 8 Aug 2021 03:57:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzvWnfBxAg1zgw46E6zGNiOzZmxo+0BnbLVqUKcRszp9C6qewT9mLGmEKLfmIxHmGSCJi2X X-Received: by 2002:a05:6402:30bb:: with SMTP id df27mr23870650edb.186.1628420227506; Sun, 08 Aug 2021 03:57:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420227; cv=none; d=google.com; s=arc-20160816; b=YKT9FZjCr4NlQ50XEfz97hZcdwUjxMiu9VMKyfCQ0B8vyI7E1XeMyRCCRcTOHcFmvP DLTWGakn163LTG0u+5LwoSypjucPaIur8bxE7hbNZxDZEjmicn2/kIbtuS82m++Z5q4G CEknZDMWd8xaQ7BWYgnt75CmU/b2gKZgRFOrUQLcstFvnK6r0KtlUVIhiTB7po7f6GQJ ZtjlcpeRT/7oN/ixnwMgQcaHe62ciqV+aT90esfkmzfJ2Aml5cwbRmyMLG/vPvEpQCUQ GxL2GqZREjXDwQ/ut8gO9sVXOtbnqt8zEOgfiUrtQW2zuXqRWwb9GxoiGOyeVfLzJq0j 8z3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=f+WSFKBA9FzXE+IUBoApd7lWgIwP06DRkt0a8gZ2ZCw=; b=RLyDwdaShPE3TopLneTUg4QKSh7qB8fda75uK1wWFQwV89g/uR0cDxkaoP2+FltBf8 OKAJ/XAF8Qnotb17AkiRuSo3EpQiXlb6JeYugrzUlelzbck4vP+4mPT3ACW1YnSV45IJ dPcI+7i9Ho/llBwivEKzlnpXzSxWp/Y+CBYZ+bQWzK0YNubzvnbGAtdblDqpjPqjWCaa AJDd4Qn2dgu/eEPpA0FwlfkSzq3xFtn533Tv5dzugS8Bx6m9W5sik3Xj9F91xjMYxNca XwIZwhT/HDPeXS/U3qGkoCako/3T7EJLqd9+rpMLMVdPRKIqt2/h8hYxOJV4jy4gWDIZ YAMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=Tf82Jgj4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 18si14996525ejj.476.2021.08.08.03.57.07; Sun, 08 Aug 2021 03:57:07 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=Tf82Jgj4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1037B68A4D1; Sun, 8 Aug 2021 13:56:24 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 109EE68A4C2 for ; Sun, 8 Aug 2021 13:56:18 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id u2so13315840plg.10 for ; Sun, 08 Aug 2021 03:56:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aIC+ys4Ka+f4jwa/BCqleqVXbU7cT/w7k8QSQ04uLvA=; b=Tf82Jgj4MKFqJkRb75BBUaqgwXAiG07kcEdsTZPMLVzDIXE6SC8A2B026t1j3IeiMh yJJS+GlTI2PvVtSzB6bpIyik8GQBOewEujq4ucJXmobAV2Nd1fZHzYa5e87tsfdmCBJd CjjlthYetLD6LZYPMV7nGohq5Kv85prrmip1rWBBBvS7+xq3hnrS1Esaj8fliwZafJH5 nR32Yr0KuQ8bRDFjkmkHGaNhp9laLHWZVWh4nMpI1hXnkZjsfg2yaM179KR3YW8zYbtk ytSDVF2UrLw/YNuCT4PLC5SJbd2dVi/7J+3lJGb7iBMEnqT1JE0oyds8OBbl2xidnvY4 BuWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aIC+ys4Ka+f4jwa/BCqleqVXbU7cT/w7k8QSQ04uLvA=; b=M51V1MOlCgXZ0JbDMdvywLn+6GNuCewhq5rfPtv/pAVCv7/sUSlDwPt7dJZtFqTs30 PnL2s4Az/BfWN2SUzrwSldeozWCIavLkvh/dwiZ4uBfssJGuHTaG6iNtk3VW3HntBNIN Ddo30VaM9rMF/9FSkIOpWXeBZafBOEJ39ghu6S3vZONvw7DIZe1JI6ZYC5mvMSlUAHFT jZx6I8UUc0iJJY1n6g+B0udHsKbqpmFX7ZXuxTVgIlnJaYcskcLsaqJyykaXbTj4xUkY 885YVebKDllv0PzSwQfik4wFPerp2Tf21vAa4aocjCqJHn+f/CkNw2KFgOdv1W/ljxkS m2mg== X-Gm-Message-State: AOAM532r5lmdR8Ar+hM88SBtYCV2iBc9I2Zu+pUtmyeF1qgp4s9TSAo9 gq2cHUaRt+j1rTbUwxWP7o59bgbmoPsq/w== X-Received: by 2002:a63:b950:: with SMTP id v16mr444343pgo.328.1628420176309; Sun, 08 Aug 2021 03:56:16 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:16 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:36 +0530 Message-Id: <20210808105539.18822-6-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 6/9] [GSoC] lavfi/dnn_backend_tf: Add TF_Status to TFRequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: bQm0fpnGdBAa Since requests are running in parallel, there is inconsistency in the status of the execution. To resolve it, we avoid using mutex as it would result in single TF_Session running at a time. So add TF_Status to the TFRequestItem Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index ad5f1e633e..935a2618a1 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -75,6 +75,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; + TF_Status *status; DNNAsyncExecModule exec_module; } TFRequestItem; @@ -165,9 +166,9 @@ static DNNReturnType tf_start_inference(void *args) infer_request->tf_input, &infer_request->input_tensor, 1, infer_request->tf_outputs, infer_request->output_tensors, task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + request->status); + if (TF_GetCode(request->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); return DNN_ERROR; } return DNN_SUCCESS; @@ -187,6 +188,7 @@ static inline void destroy_request_item(TFRequestItem **arg) { tf_free_request(request->infer_request); av_freep(&request->infer_request); av_freep(&request->inference); + TF_DeleteStatus(request->status); ff_dnn_async_module_cleanup(&request->exec_module); av_freep(arg); } @@ -906,6 +908,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->status = TF_NewStatus(); item->exec_module.start_inference = &tf_start_inference; item->exec_module.callback = &infer_completion_callback; item->exec_module.args = item; From patchwork Sun Aug 8 10:55:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29348 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981310ioh; Sun, 8 Aug 2021 03:57:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJziY+D0zVyoUi2TZ4LZqQQHphj8gdAJTVg3tPr281O7XsBaKIIQY9QHDnympmskgV37HxxY X-Received: by 2002:aa7:da02:: with SMTP id r2mr23566305eds.249.1628420246772; Sun, 08 Aug 2021 03:57:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420246; cv=none; d=google.com; s=arc-20160816; b=Xi2iZ3kqKQH/sbjoFawz0H8o/ZWIAy1e5SX7cP+/PNE8Lmf1wJWs7Bj0Q39+loYGE6 iuAb/nRlD5sFjQqDzwfpGPnYhjEkKDo49pvWbq5sui0c8gto0KXPXqx22WLzTzB9pM9u qq3+3FkVXvEFkWPfPkvcQoCRczQGiEieMN9ZYNE0n6hxH4ilJuKsAFo4PZtbPLnI7w+R NSDeC6GM7xd8dKpxcrkv7rQ+DpjDAcG370/kzYA7fo1+uWhrl4OAep/X6cR3EEkMY6Oy J3y4ST2d2Z+7ZCHotCsiQcWQeDMm/oR+K4aQYWb61k4suDJ0xxD9KR3DyVpx5XLYjURB GS1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=OgUuaP8xPpRGK6OrUayodP2ww1RTim5e3wbbMR4gbAE=; b=WP753+pASatWdZr7BHOeuH4wI3pYzAn7DAyTL2uDwHOgn2J9zcyr0O63gENJecCA9v Fy/hCnrImpITjh3VVfzU0VyvWfDqAHFGQKP+B8i3cXxZUeU/v5WDVF7ZweXOHECFSq4W td6hgQD7Vv+PH4Ex78huN8blnuBGbOqV8pq+M6e+5N7nFekqM/WYda9zSwW8iLQ9Ag/D sckBPtTHTfoxec41op3w/nMokhqQWQVQBExq8LhIDKNCi1ScMTuqPJH+dmbaXFEDPzQ6 e5a9A+uHsykt9Kx1LGZxx26qUe0np07URuvEXdgL6tuXznN3DlTWq7vZA+KVQHRyz6wX 6q9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=k43iN1jm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id cq4si14883876edb.74.2021.08.08.03.57.26; Sun, 08 Aug 2021 03:57:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=k43iN1jm; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3E6FD68A52D; Sun, 8 Aug 2021 13:56:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0F91168A441 for ; Sun, 8 Aug 2021 13:56:20 +0300 (EEST) Received: by mail-pl1-f182.google.com with SMTP id f3so2716906plg.3 for ; Sun, 08 Aug 2021 03:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N+koUDzWtuSowqPYVU7R/lvjG5E1OjXVK6thHCxRP2E=; b=k43iN1jmjOPJJ4lc7zdC4R+qllD5EKXXOBnXvJPdct5QI7dMUwkhAAoUwOkM3uZUGp fsUS1uZOY6sBZt7N6/mzStQPSbP+Zfdm04yPIopfX3Lf4xSghve1Pgrr8eJYaLLHIJfF RmNg+S5C8Kto3akIiFy7sZtfr16F/QvBI6eaUUUXLC8r1mEmwE9O5HTVL/C4iMlEVCfa vDJRBhUAn070F+Z7c5M7Ub/8RGVWOKSJSdD7g0Uy8WMQOCGg1oFiEcrGBaFxeLI48kVJ B+DckISkbvtebm8HHwOGVmeWt53ugiXTahvUzhHhCYSg+aYOhTaknYOnypbwkb2CToY4 RPpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N+koUDzWtuSowqPYVU7R/lvjG5E1OjXVK6thHCxRP2E=; b=TMAT2o8dbgtagNY9tN2zudOB9EOTxVBFvGW5j2P6QzcaiWV4Xclicvk+bLKbGJqCt3 xFDGq4+Gm8G8CGR0zrfe1lDUzQJVbhr8BrSQhbdNIfo3EZpxETwAmrvdx0CZ1cIcCGIj ZsOOcWPjjUU9R71v9+ta8wDl5jnHG8MExQYdKHjCrTD9kcVd5Zlt5hAXrtt9YOFuCHUX QTt8kJMt8tIAvFqZC+61IBcdMlcK5wApz5cljw/iuj7U0MeVBdMZcyjwf8XBCai2uCsk vH0F9OTOAusRRaiYUvj8+x6n2rMVrutjbcr/YuX9BXcBZirsWOGK+jGk1urFoBJF43ao Agtw== X-Gm-Message-State: AOAM5304Md9pd13Q+/3CLRzSBqWJRk4eIAEFyeDnNzROJTwUgf6+m3eT 86v2ysBw2ksuT5kvaGHhj9WCtEESPEJgdQ== X-Received: by 2002:a17:90b:3810:: with SMTP id mq16mr22236301pjb.168.1628420178273; Sun, 08 Aug 2021 03:56:18 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:17 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:37 +0530 Message-Id: <20210808105539.18822-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 7/9] [GSoC] lavfi/dnn: Extract Common Parts from get_output functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: z1bS6HnGOzZD The frame allocation and filling the TaskItem with execution parameters is common in the three backends. This commit shifts this logic to dnn_backend_common. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 26 +++++++++++++ libavfilter/dnn/dnn_backend_common.h | 16 ++++++++ libavfilter/dnn/dnn_backend_openvino.c | 51 +++++++++----------------- libavfilter/dnn/dnn_backend_tf.c | 42 +++++++-------------- 4 files changed, 72 insertions(+), 63 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index df4bab85e0..470fffa2ae 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -142,3 +142,29 @@ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **i return DAST_SUCCESS; } + +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx) +{ + AVFrame *in_frame = NULL; + AVFrame *out_frame = NULL; + + in_frame = av_frame_alloc(); + if (!in_frame) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + return DNN_ERROR; + } + + out_frame = av_frame_alloc(); + if (!out_frame) { + av_frame_free(&in_frame); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); + return DNN_ERROR; + } + + in_frame->width = input_width; + in_frame->height = input_height; + exec_params->in_frame = in_frame; + exec_params->out_frame = out_frame; + + return ff_dnn_fill_task(task, exec_params, backend_model, 0, 0); +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index e1d678b230..604c1d3bd7 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -137,4 +137,20 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ */ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); +/** + * Allocate input and output frames and fill the Task + * with execution parameters. + * + * @param task pointer to the allocated task + * @param exec_params pointer to execution parameters + * @param backend_model void pointer to the backend model + * @param input_height height of input frame + * @param input_width width of input frame + * @param ctx pointer to the backend context + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if allocation fails + */ +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 47cbd48ba2..c825e70c82 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -640,10 +640,15 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu OVContext *ctx = &ov_model->ctx; TaskItem task; OVRequestItem *request; - AVFrame *in_frame = NULL; - AVFrame *out_frame = NULL; IEStatusCode status; input_shapes_t input_shapes; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; if (ov_model->model->func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); @@ -669,51 +674,29 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu } } - in_frame = av_frame_alloc(); - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, ov_model, input_height, input_width, ctx) != DNN_SUCCESS) { return DNN_ERROR; } - in_frame->width = input_width; - in_frame->height = input_height; - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; - } - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.nb_output = 1; - task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } ret = execute_model_ov(request, ov_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; - - av_frame_free(&out_frame); - av_frame_free(&in_frame); + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; +err: + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 935a2618a1..3dfadf888a 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -310,36 +310,20 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu DNNReturnType ret; TFModel *tf_model = model; TFContext *ctx = &tf_model->ctx; - AVFrame *in_frame = av_frame_alloc(); - AVFrame *out_frame = NULL; TaskItem task; TFRequestItem *request; - - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - ret = DNN_ERROR; - goto err; - } - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - ret = DNN_ERROR; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; + + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, tf_model, input_height, input_width, ctx) != DNN_SUCCESS) { goto err; } - in_frame->width = input_width; - in_frame->height = input_height; - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.model = tf_model; - task.nb_output = 1; - if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); ret = DNN_ERROR; @@ -354,12 +338,12 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu } ret = execute_model_tf(request, tf_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; err: - av_frame_free(&out_frame); - av_frame_free(&in_frame); + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } From patchwork Sun Aug 8 10:55:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29342 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981377ioh; Sun, 8 Aug 2021 03:57:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxupM5qeW88qLJVikSrTXzj9aJBsCw36J+BmWuw2U/ZHhdfCSdBEm0s9Voaghk2wf7tPxm9 X-Received: by 2002:a17:906:c085:: with SMTP id f5mr18271295ejz.250.1628420256402; Sun, 08 Aug 2021 03:57:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420256; cv=none; d=google.com; s=arc-20160816; b=QWUL3kz92i7nilMGmTjbm5/lDUgYki+SnJQa6/MgWKGMhLG/kNmf66OMSUwlgCqxMh wnx8spkgnllW4K9dZjb7VzRssf8rYHMuRPeXhqxiO0cWzBhkk9kBtkinc4e7Zitl6yvb 10BeVfevVjUq7PtDLkI4iu8gPBV7ohHzuvjq+I3dnyB2UCFogR6aapekaOVepz2u+G5Y mtVw9XJsu2sHaVNRmGupbfgyX0FVXMOkz1STa4crE+TRKCALvXn8FL9+WzgY1fsxnJQC HwCSMiF+Lg2r86cSsxfXKimadhlERNUxSyMM/4GU9FebFS7IgL35scxrbfRmixF1Zd99 GX1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=ehiH8NuZLqs0qyTPWmm6DPr3zKk4nMDwauByNoznMig=; b=RKwqA0gV63/s+4OfYbIB0eG0ftsHvRBO1acN0cdK3DHYqOYa0N03aQs2NSIhiAnJxF Qm8wRhtI6hEqnJgsNYwrTP+TqVMGnVDER+c3NFwCZAANWM9R1UVUaoFyckVgtr1aAueg UuhQ/BAV8N6t2+UYr5hhvbdYmRKDDev+JxZ2OLfMpQSQdwjDT2Ri9halQSI5w+28q7Qo O02GG+0etog7+IVH5CgQgBWAfidLPD5auEpcl1tI1ZdUuUd8y/jFOZDsLA1RX33sJmrx pQui2O2hJlHEr3531x9UDOQ6fZxaB8G7Vz95LbdUbbJ7XjxIpBZtVVFiA457uYl67/W1 gcLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QvxmVHyz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j2si15409535edr.294.2021.08.08.03.57.36; Sun, 08 Aug 2021 03:57:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QvxmVHyz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 286A168A538; Sun, 8 Aug 2021 13:56:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B030068A501 for ; Sun, 8 Aug 2021 13:56:21 +0300 (EEST) Received: by mail-pj1-f46.google.com with SMTP id u5-20020a17090ae005b029017842fe8f82so15926037pjy.0 for ; Sun, 08 Aug 2021 03:56:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6ndOetGV9cSoWxHpy78+dRRqH/fjSVTbeMiB8ZD6+Ns=; b=QvxmVHyzRaQhEWG/vec4LHoH6QgvZn6FEKsbDnDwAVkXiWIFqvTLNPPCZjwEOt8b60 ge28ce2kl1e+ZiDNFLaoe5zy5v9Xeibjv7LksVL7s87Nsg/PXbYqctfq+PSZ/V6CPX3r nzZoTT+boLtQx1wFAuXUFN0y0/uOt87zzE1Wp1uwgQVNF7BdNe582LOim/aG+wXOkUF1 1GyGS9tqdJe39eQnS5oMdnbNgkRfjpY0OsZo2WnDZVZVmmw8AxNgQ/YU2mHPda7kPAii l/viearO22ueqZPi1A9ZMZsavRjaKldds3Rai8/+/tWDqcINxog4QhttKtQsF6++YLMq 6Fkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6ndOetGV9cSoWxHpy78+dRRqH/fjSVTbeMiB8ZD6+Ns=; b=m4FDEPm9vgYb9auFBWitOA3cF+gvHRhMyxM07Y0IXyzWikSijRtOTFfYSOzPIqC0vk Lsh8w8j1GwGaUl6kWXQTvs3aZs6gRcUL/PDmundWWST0Ox5UDYC2ciILZbhM4n5YlTw+ HWpMQrZ6XFOzPAVEn9T2x9FKacEF+qx3uOaeN3HmOD4qJufAIqAgd6qjlKXJXT5X8s5L rIWEk3Td4wgI+fJYfl9A36acGerdgcCctKvMcbV/QLgPjKH6invgpk+cD1gaF3dE1e4G Ftar4bQ1sOL52Ir+c9qy82B59SeZqeJaxTCCxHJ2W8XJTQ+WgyVpQ3wuLJFXXWkDF8Ap 6h+A== X-Gm-Message-State: AOAM531MI0e08iMpX4/AI4y2S1bhZzfh9DFNF3EHKiU+YHThgB9sQZzn RDhg7PALIs160EORP9cW9UpJWINmEk2n9g== X-Received: by 2002:a17:90a:ead3:: with SMTP id ev19mr19387491pjb.229.1628420180006; Sun, 08 Aug 2021 03:56:20 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:19 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:38 +0530 Message-Id: <20210808105539.18822-8-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 8/9] [GSoC] lavfi/dnn_backend_tf: Error Handling for tf_create_inference_request X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OAf46GtXAXdS This commit includes the check for the case when the newly created TFInferRequest is NULL. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 3dfadf888a..fb3f6f5ea6 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -135,6 +135,9 @@ static void tf_free_request(TFInferRequest *request) static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); + if (!infer_request) { + return NULL; + } infer_request->tf_outputs = NULL; infer_request->tf_input = NULL; infer_request->input_tensor = NULL; From patchwork Sun Aug 8 10:55:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29345 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1981246ioh; Sun, 8 Aug 2021 03:57:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzbnZhlwXrcy9tPjMqWCzfUNYJUfdpwx8Dyy7tOTsJla6az4P1DOVJb/b2pcV5ytArLuifu X-Received: by 2002:a05:6402:4245:: with SMTP id g5mr12562318edb.222.1628420237044; Sun, 08 Aug 2021 03:57:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420237; cv=none; d=google.com; s=arc-20160816; b=QdrxZo0dOeAgYxycHOjyrSdfZJhJY7g4MHq1vlDVVgC/P3/U5aoxw9UNTQUQ0RatjW UTD2Ql3KvoaS3G1Zixg1e8H+92Y1YPlRvVe+gw7JrnKe/nABEH/DcWPZb6JGKPs3VdxM 9cCrkhCkzTC32TRyti4jQ9qSyZG8B7vZPci/7+ZBogx4/5A/Q/8AKxdsO9NQqfAd6FwL ncz3P3OFZ2xw+ZiVcRsm9RHCKuof7EpuPyw6hVO85/Hk++79vtiiJdK7cZ1IRo3SBiGP 2chWcjZs318sPhgX5Yg12GHiYtTKbgg/FnK7JM301Zj4tVupH9Tb0tCndwZRBZdgbmTO EhIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=HcVENjH8zUeXJTUVgoQE/5nwBxCzobspYb/vgB82lyU=; b=HX3UKc9E/DBopNyI9pDmZyN3OqLeU7fAjg61fsvL7yPzflzhUlS57k7bHp7zxLqK7b ZBLD5VBcf1cftnnm1sHs1DB6LNEqomS91zx3ivqP6z7/NZBZb5dnahJqAdGNU0kIPf2T wvw83IxxZjG7pN9u83+5z+LZ5zBZBfPUFC5f1jT/Gg+gznscTf+4Y+6rLUrBBBJ8PKZM YpuQc3DG3kXI7R7LyMA23j/zMIwgYGmvs3M84nLaUw9X9Y/QR04QuRGuPvF9e1PCapx9 s5xdyRXUf9xxpJ21izPnY/SFhkmTtmw3Q0HLYqmkI+1v262YiRpWhtpwuwqYwl4Y/KEI DV0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=ZcWhVAZR; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e22si14173209edu.306.2021.08.08.03.57.16; Sun, 08 Aug 2021 03:57:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=ZcWhVAZR; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3DCD368A4C5; Sun, 8 Aug 2021 13:56:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B40B868A4A4 for ; Sun, 8 Aug 2021 13:56:23 +0300 (EEST) Received: by mail-pj1-f49.google.com with SMTP id u5-20020a17090ae005b029017842fe8f82so15926078pjy.0 for ; Sun, 08 Aug 2021 03:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LRTsiIgOHS2t3TDJQ4+CMx0Hal6WW7XWDN4gDxtyPdE=; b=ZcWhVAZRyuBJxULWNZK8K86Sspviyk+dBv3OTQMVfjsIlvggxujJnjShm4WazyjlT7 h4EzLEuU2Z5OuDQdQXWTg3bewnS8ERwJNFTrxUv6eelJ8xJl4Jt08y1SvlyKuI4sUHM+ yr7qfvWM5UBGZNxsn0fCaD5A/BZN9k6l05a6n1QOUlJ9Ot3bzajtaLvDcaJo1G2zYsR2 Zc+Xb7Jcmef1uvdVP2cd4VwnoRlen/tDxl9rrclsiciBohOdacoR21sFvQP9HQvnsECJ bvUOPpsEnqCzgi8mx7zyWNDmeZYN5xXVDVe/y2sUrHC8zx/NgSeoQU5mzP2tWoD45wG0 a9iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LRTsiIgOHS2t3TDJQ4+CMx0Hal6WW7XWDN4gDxtyPdE=; b=cOKMFI1CCeTTWzzeaehdJgm10HYFPnaIMEY9nMQfqF9zOnc24eBFknF5nkzSJ5rDiI atEYWaGhfJGGu4phY/KvoriU0onRNRfAnLg5+3bQd4Pa6SPlCwcaU7v5op70i19Vwchh +PZfMzLlpMmYJNQB15IxOXRQMqnifElup8ssGK6DUBZCZIQD6/t1c9V28KAeQM26RWC6 xir3XlH4S2CaS7s+p2OKJdzExXaSC07JRXsR6WtFczWAe4Ne1Apj7N+TVfCN9gwNpHQe 4OzI6ha/a2jZSMCEHKywBZcV+cz353ZaW30zrq971L8ALywxFb5eXrLAfnPrk0cc5zqs JGGA== X-Gm-Message-State: AOAM533r48qJZG0iU7Sc23A1W1/+l5sGLCaQC2dgUTW4m29KMi9dzAlY EFhpcvJQLwWC0Fz18RcrtTUFamTb1tLLLQ== X-Received: by 2002:a17:90b:400e:: with SMTP id ie14mr30587692pjb.156.1628420181731; Sun, 08 Aug 2021 03:56:21 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:21 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:39 +0530 Message-Id: <20210808105539.18822-9-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210808105539.18822-1-shubhanshu.e01@gmail.com> References: <20210808105539.18822-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 9/9] [GSoC] lavfi/dnn: DNNAsyncExecModule Execution Failure Handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: y1jVnkVB4xPz This commit adds the case handling if the asynchronous execution of a request fails by checking the exit status of the thread when joining before starting another execution. On failure, it does the cleanup as well. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 23 +++++++++++++++++++---- libavfilter/dnn/dnn_backend_tf.c | 10 +++++++++- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 470fffa2ae..426683b73d 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -23,6 +23,9 @@ #include "dnn_backend_common.h" +#define DNN_ASYNC_SUCCESS (void *)0 +#define DNN_ASYNC_FAIL (void *)-1 + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params) { if (!exec_params) { @@ -79,18 +82,25 @@ static void *async_thread_routine(void *args) DNNAsyncExecModule *async_module = args; void *request = async_module->args; - async_module->start_inference(request); + if (async_module->start_inference(request) != DNN_SUCCESS) { + return DNN_ASYNC_FAIL; + } async_module->callback(request); - return NULL; + return DNN_ASYNC_SUCCESS; } DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) { + void *status = 0; if (!async_module) { return DNN_ERROR; } #if HAVE_PTHREAD_CANCEL - pthread_join(async_module->thread_id, NULL); + pthread_join(async_module->thread_id, &status); + if (status == DNN_ASYNC_FAIL) { + av_log(NULL, AV_LOG_ERROR, "Last Inference Failed.\n"); + return DNN_ERROR; + } #endif async_module->start_inference = NULL; async_module->callback = NULL; @@ -101,6 +111,7 @@ DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) { int ret; + void *status = 0; if (!async_module) { av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); @@ -108,7 +119,11 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ } #if HAVE_PTHREAD_CANCEL - pthread_join(async_module->thread_id, NULL); + pthread_join(async_module->thread_id, &status); + if (status == DNN_ASYNC_FAIL) { + av_log(ctx, AV_LOG_ERROR, "Unable to start inference as previous inference failed.\n"); + return DNN_ERROR; + } ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); if (ret != 0) { av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index fb3f6f5ea6..ffec1b1328 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -91,6 +91,7 @@ AVFILTER_DEFINE_CLASS(dnn_tensorflow); static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue); static void infer_completion_callback(void *args); +static inline void destroy_request_item(TFRequestItem **arg); static void free_buffer(void *data, size_t length) { @@ -172,6 +173,10 @@ static DNNReturnType tf_start_inference(void *args) request->status); if (TF_GetCode(request->status) != TF_OK) { av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); + tf_free_request(infer_request); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + destroy_request_item(&request); + } return DNN_ERROR; } return DNN_SUCCESS; @@ -1095,7 +1100,10 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q } if (task->async) { - return ff_dnn_start_inference_async(ctx, &request->exec_module); + if (ff_dnn_start_inference_async(ctx, &request->exec_module) != DNN_SUCCESS) { + goto err; + } + return DNN_SUCCESS; } else { if (tf_start_inference(request) != DNN_SUCCESS) { goto err;