From patchwork Sat Jul 31 05:35:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29151 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp145748ios; Fri, 30 Jul 2021 23:02:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzoxo9BNpvvOLS/0g8wqxV3ucqWu3GXLscFeuALXhoor2++btTBQiD4xC1GwPjGhwr1WzPJ X-Received: by 2002:aa7:cb19:: with SMTP id s25mr7595879edt.194.1627711374860; Fri, 30 Jul 2021 23:02:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627711374; cv=none; d=google.com; s=arc-20160816; b=upOrd/jz7ATC4JIBVX25kEkcvR/IkoPEGlE8jP1qgSXattsw3V5UOOcSfh9gU/0pgV yfwSVhEzIPUVZKMBOB3DlQ6Uc5oKYNT1D92XQMCXiWVTekafbxYk9nQILIIn9b26FTMl JwC1Y1McmbbJQRKw1iGwdJ0tVTDa/Q4MgSwDkJukipdmslUlwgSK0Tj1tS2iVIVf45i7 3T7Kn5LcQJnOg4ffuFEMKRwKLNLInTSbHdhRLL/GK+6TfK6K2qAVqUacbyx7rVVj+c3G 7tRiCzHASi949qriD5L/GS6mlTtMtC5EEe81LE6CpVmCjgbpNRFrifOe+VA5dCyAuoe6 qo5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=VnwPfeuGovETQ8gZg72Ivr0N5stiEFkD54JuqY5bBfLl4lzUDFDsnKmXe+gaBtnx4x d4R64aAVEsGe50AL1GeF3oJPT9YxUeWeZyytRXNcWgwZFrTS+Z4wwkuexROONXGSNvXW vCxpeaoc/RC6xMekPIdGD49ZGNMYBtyGPp1Xqv01xSN2sFfIgdwppHEUcw40xyiaN7k6 HQ6MHkuUMIjM2/tRfnsWEKADbtWj5ozFVZqA6I5m1/CNFdVAp1iTl853dd7UU0zs/VI/ TB7khx/JtkYpV8KpFJmI1BeCMUlEjtcFC4N8H0V6Hc8EUC3uEFkJabcKK6407dq4G+Wj 9flg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WrdUt6of; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bl4si1714786ejb.689.2021.07.30.23.02.54; Fri, 30 Jul 2021 23:02:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WrdUt6of; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 788B068A466; Sat, 31 Jul 2021 09:02:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 79F24689F1C for ; Sat, 31 Jul 2021 09:02:45 +0300 (EEST) Received: by mail-oi1-f180.google.com with SMTP id n16so9803883oij.2 for ; Fri, 30 Jul 2021 23:02:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=WrdUt6ofGeKK/SPj+dFPy4bjSvtTGI1ffL28EYruF506/FlMVrdqG1OffGAS9e6brJ 0CpV8b2W8Kqa2YhXP1UoHvMcdrb60CQvaysw3tw66WSj/am2LZv3IsDwC+7+N0muzdaW 6DNBRYoWSegWJpEylyS6CCaXH6XuUifrZmVqDQWm8gLzoB/uCQCcnwXDpdEDCVTlQ20A azZSJDESGxXqRWPL3RBP+mykEzrnmNvpX+PCLdU0p1Nw5CdAimMaYRrqQ2HHlvQDYfkH k+0vZNYPrjL401nZGYetkg3j5yNGM0adZMfG6vCuAOp4gXgCr7MNQ1zJMfxekPq2JUB0 8cxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=Cis30iefcI468yuM1NSy76vwnV4WKMa/BrB31G9uW+RS4fAppmRQ/zQSva64P8vLw8 y89R3mTVRlkm0BVnzcOn6UloNWB1vdIl3xuhZZnnsaSuE00oYEro7QBcWa5TMPfIb87w n8BeoAylyOqRqwUWSO0ZRyT+AFjnooK4zUY8C5JGFPwzNwwCXKIHnQmdKWTWntaJ+J4i YS2B5GHyCHxnQnX0l8L6E6FS5s3NiWoHH1o6CzRGrke8vsUylJq0l4HtC3Z6Al6nwOoO HEbUl+OvzZ1BPCcj5TX/NEmyD3ZRfbXTPm6JdKxhgc4nk9fPNhXuzFVaPil1Uf1UVtmD BtXQ== X-Gm-Message-State: AOAM533Dl0re8k/ydCPlklBw2JJZ0jKe5ANHdqGHXGDtqGDVDShtYxPW lepnTtMbt1KvtQdfG8LZLBzIdselTANPxA== X-Received: by 2002:a17:90a:7109:: with SMTP id h9mr5640834pjk.35.1627709756225; Fri, 30 Jul 2021 22:35:56 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.35.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:35:55 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:09 +0530 Message-Id: <20210731053516.7700-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/8] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0I97kkb3YBoH This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif From patchwork Sat Jul 31 05:35:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29146 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp136224ios; Fri, 30 Jul 2021 22:44:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyE8bz69y2krlyhLiDVBCrYtUJcIZH4Ud7LpX9M8/hfIelnPNiKrst0mjdgb8vUFZGsjA4l X-Received: by 2002:a50:9e82:: with SMTP id a2mr2941240edf.115.1627710257579; Fri, 30 Jul 2021 22:44:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627710257; cv=none; d=google.com; s=arc-20160816; b=LtEuchBGg7cMaWiz2NgF8v4w1CnRl2wiuF25SQVTIMQ8YJyo+xvQduv4EVaDt6vReZ BfNpjOKj9mGK0cyQn5L+6spMDo1z0eEItigf1ESSfSPbiKQ3DpUZRDrZY70jTY8O4S0h 9S943jWZm3/QNgvGI3GHCd6Nq3jjGb5k9xDpqmKUsVdW3S1X8FL5FKA9DCCgvOFBj3+5 LzlcQS7Ftf/kWVZyq6YyT9gnNDEZaLRzezYPkTvwcfOhAgQzjErwLNtTXLK7hdDV1jdh 6C7lpzkpE9/TJknF+Nf636IjFcY/szPAR+9SnIAI3Ny6b9SwtNvHit7wzdC1HmfY/y0E 7k/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=2BaZt/nZ8tYkB2dMxu8/Sj+l4fN6MVIaca0p0tqv2lg=; b=I65SZxOG9O4981kh42yPiPL7zuk8MPZNgN1FtMG1UD6+I31Lf1KODXXVIC6j89U/hu Z4qUDf0UvSHdwEea2l0S3Z+U02enTa+X+BmSgwZnDKstAPTHzcoQwyhqkxDpN49Dqbzr 38Z6oOS41ayxJn1YzcVfMZd9YCcA37ggp3LOfIoq+OwrBfifRhkYbyQbCODMhnY1q5KY p7Feyjb3nGa/YSOSVxOLBmxWXHIVvahtUKWQNtzW33NfO07AhOiV0buN7rHNdPziWxJa cjs7FxFglPKPBKxlOB8oFO+dAaxK6HXZnS3DAm5kFAKHnIbVk1Efv3Is494s8lY/1z33 WIww== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=nwdG79GO; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j25si3773269ejy.93.2021.07.30.22.44.17; Fri, 30 Jul 2021 22:44:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=nwdG79GO; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DF4E168A3D3; Sat, 31 Jul 2021 08:44:14 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 326E6680210 for ; Sat, 31 Jul 2021 08:44:12 +0300 (EEST) Received: by mail-pl1-f175.google.com with SMTP id i10so13580055pla.3 for ; Fri, 30 Jul 2021 22:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=nwdG79GOtGIyLM/UK4mmD/cQRG7WNuj3TkbYMyvjNHGenNhAitl3dISzQpwupWmdxy 1zuNoHml48Sn2Q1Va0/V2bOGCiGhKsJl+vrOXVhV8UOUNri+TinsYZ9A9Q09JvEQ3MOX opI2tfp8G3JyaJ2SH61p6trso2OX/xSNgDdJoxMDwKUeLR6cRZ2jVgZ/rhY0SaGCpVTR Wrt0WQDJs6of7oCSAfQWVTzR4VHoTh5hRXIDWMfNgFgSiA7SANVKUpkCAa0MwdytJsnI niPZMUl4o7c17NKEzsW15lv6S25bRdyPbObWsoyQBsHt6ww68zISri+tRScWt9ysfNDV H+Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=kBKl49Jwq5ENci+1Q38KbeHdo1WHPfyM0G8XvPXRCAoRBq+7S69dVKXTRWks+ZfkpI dXh90anWaKeAcd8iDJdlyghbEHUdT+VafdGkCnj5P/7xE6RzfG7+aNX3v3AVKS6YHleC Yq3SCJNV2sbu7xItUz6/YYvwfsvbYDHAXJbDFD1CbJHxnNCcTEh8CnGBtiKj/lBJU/EQ 6Y1ebnLb2hd4l/ZRZRfX9JExm05zRGbIq4eWDMMPovQK9+RWjUuRIy2jrHOGihXlWPXg 0BsELniVtNxbh5MoTVBW5ySf7M0N+RyUN9avkV6IR7Ti8M+kqDoWrNU7XCBVM9YhihTM n1Hw== X-Gm-Message-State: AOAM533R12bnjpoBYXvdC8wM7Z95++X4P6WPDCs/sONuS12lgnHZV1ML LI63Tab2h13v/QLoh477hsfmjvzDorP2jg== X-Received: by 2002:a17:902:c389:b029:12c:a310:23d1 with SMTP id g9-20020a170902c389b029012ca31023d1mr412960plg.66.1627709758012; Fri, 30 Jul 2021 22:35:58 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.35.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:35:57 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:10 +0530 Message-Id: <20210731053516.7700-2-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/8] lavfi/dnn: Common Function to Get Async Result in DNN Backends X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: dHybB6CZ+1oJ This commits refactors the get async result function for common use in all three backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 20 ++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++ libavfilter/dnn/dnn_backend_openvino.c | 18 +----------------- libavfilter/dnn/queue.h | 1 + 4 files changed, 37 insertions(+), 17 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4c9045501f..df4bab85e0 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -122,3 +122,23 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ #endif return DNN_SUCCESS; } + +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out) +{ + TaskItem *task = ff_queue_peek_front(task_queue); + + if (!task) { + return DAST_EMPTY_QUEUE; + } + + if (task->inference_done != task->inference_todo) { + return DAST_NOT_READY; + } + + *in = task->in_frame; + *out = task->out_frame; + ff_queue_pop_front(task_queue); + av_freep(&task); + + return DAST_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 96e4df9676..e1d678b230 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -24,6 +24,7 @@ #ifndef AVFILTER_DNN_DNN_BACKEND_COMMON_H #define AVFILTER_DNN_DNN_BACKEND_COMMON_H +#include "queue.h" #include "../dnn_interface.h" #include "libavutil/thread.h" @@ -122,4 +123,18 @@ DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); */ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); +/** + * Extract input and output frame from the Task Queue after + * asynchronous inference. + * + * @param task_queue pointer to the task queue of the backend + * @param in double pointer to the input frame + * @param out double pointer to the output frame + * + * @retval DAST_EMPTY_QUEUE if task queue is empty + * @retval DAST_NOT_READY if inference not completed yet. + * @retval DAST_SUCCESS if result successfully extracted + */ +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 8d23446296..47cbd48ba2 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -32,7 +32,6 @@ #include "libavutil/avstring.h" #include "libavutil/detection_bbox.h" #include "../internal.h" -#include "queue.h" #include "safe_queue.h" #include #include "dnn_backend_common.h" @@ -883,22 +882,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa DNNAsyncStatusType ff_dnn_get_async_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) { OVModel *ov_model = model->model; - TaskItem *task = ff_queue_peek_front(ov_model->task_queue); - - if (!task) { - return DAST_EMPTY_QUEUE; - } - - if (task->inference_done != task->inference_todo) { - return DAST_NOT_READY; - } - - *in = task->in_frame; - *out = task->out_frame; - ff_queue_pop_front(ov_model->task_queue); - av_freep(&task); - - return DAST_SUCCESS; + return ff_dnn_get_async_result_common(ov_model->task_queue, in, out); } DNNReturnType ff_dnn_flush_ov(const DNNModel *model) diff --git a/libavfilter/dnn/queue.h b/libavfilter/dnn/queue.h index 2524d5fa59..9ab5c89f65 100644 --- a/libavfilter/dnn/queue.h +++ b/libavfilter/dnn/queue.h @@ -18,6 +18,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #ifndef AVFILTER_DNN_QUEUE_H #define AVFILTER_DNN_QUEUE_H From patchwork Sat Jul 31 05:35:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29150 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp143702ios; Fri, 30 Jul 2021 22:59:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyJsgRAkgzXZkeii0nV25g1/FMUXX0a9mrx27CYluQ/D7V7d9JOrGuVNXwiFNjg94U3ofSv X-Received: by 2002:a17:906:86c4:: with SMTP id j4mr6207762ejy.431.1627711189758; Fri, 30 Jul 2021 22:59:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627711189; cv=none; d=google.com; s=arc-20160816; b=OC8n7o1atmdJ/6U0Q3742E5O3AmVpo929RMivmjq/WzAQpmBbzxmTrwmtDQTDfUGuR kFFiWi+oJD7eLoyEZAwwE7Q2DgkI6xw5i2y60YsS0LixuCEVAHnTzTbKjdZmpdf8TBcq zygC+zfsFxDODXycavO4SeftWCaJfXwvifnHGl5d7z136ZS+lbt5dxcxugw5jRAVoYlO Y0HEXaQuH5nGEXCny5MtoT4hcb21iXhuf7XSZiE68/HJLNq/8REkqbNMDc00MHqTnJxP bbX/nB7WxGkB+Jeya167ThBYnQbx29M3Yfud9OAHg17H3rjcS0xsQYdEq4sF/DCIVBGa Y3GQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gZq5amk4Fenj6Z1yPnJLFOm0T+gDdQEBhnvRZMDobOw=; b=EBKxssXNhlCNBnka7aEjP8AVqLcJrvW1gn76UORLEPsO2lknBnIw/4p6WedSrw68XP CI7rsj9xbOHkk4ALAwjkjF9p43pfIkt9Jgqkhpkdk7XLxPeYYA4y3LbiOQOLSMzmT9Go EAH3jPZ1/SL9b/LJwS3jVEcAt9KTBJWzb2Ovs9UK2U14M0D2nnkocX7iJAUgHqM3RXjV Z7grkalhMFaj08uKS7jCDGMxzLmY2b7DVVvJ+QY1jO1CNNpyZG6iH/6RdyOH2H5fv3Wu m8tk6PmZ+6rhBKLjdMsDk6zeZfsS4WqIxg87f96huIw9hNpYiQhiQhkhJb0qbGDsYbfD 16Pg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=hfWHOY2B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b11si3918912ejc.345.2021.07.30.22.59.49; Fri, 30 Jul 2021 22:59:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=hfWHOY2B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1424368A3BF; Sat, 31 Jul 2021 08:59:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A31126808F2 for ; Sat, 31 Jul 2021 08:59:39 +0300 (EEST) Received: by mail-pj1-f45.google.com with SMTP id mz5-20020a17090b3785b0290176ecf64922so23850039pjb.3 for ; Fri, 30 Jul 2021 22:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cmzj2KvWrvFM25vmhuO/8auwNOwo3QxtZvI7+o3Mr1g=; b=hfWHOY2B/kUX6uAXpPGQ5ES8h5aydjCxiY9mMBjgWJP+AuZMuRkQmyTlPSmPxGBj1L /6jrWnirJI4Zw6F8PGr2d4K0C8fkP4LDccx6Pqjt5Og9V/Zh5b27RVEyem7bSyMgfbCI xNdHocnGLM8zjQVCKo/pLDi6atNSIjiOHf8ag5mwxHoCsEAm2Uzd3Am8xR574/n+3zLG qZ0Kw/3Ul27qxOyF+fU212XXEXGfoWLqw6mP3NlpKdEkAjLJu+UGSZcPBjZkLaAWv0Kj QZlEtuGHdkiayT5Bf60FOxYsOR1dqISXLeN/xrGoseQfkRhVqDwEmEOI5ZtamZ1xI6gE oRBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cmzj2KvWrvFM25vmhuO/8auwNOwo3QxtZvI7+o3Mr1g=; b=UQ9pjO+V87GbGoKutDt7oRfQsVytq9uAdYO8C71Z9oaBNHTTMJpd3k318iH4euG4Ck Az5W/G+BokFwCym53iKpSmqYyEgaXcQNbwU3U6sM9OKhahHo/xmX0hMts6llUzPXH3Lq hL+PrBp8/lgxwPDtUHbphpphB+MDDSM4tCNPTkvKO0EMOwiPGrvNfjSKuPtoHbzmgDdl Ot5mm4f116fKU/yumTuB2TMnr1PosHOMJmYwmJs6WS9mxklIfPFMweYir4JwnRKmDoVN Z67Z971xZxipxrRLj+s2r3nd0YW7OOzpKq7xXFYrookaBzhlqoKsD/VXVJ8Nk5zu3UBR o+jA== X-Gm-Message-State: AOAM532YZVCKh6QheY8ajx0HgPZoZQciXIp2c+EpsQ0jwTGF96ZW4MMS TUvvXG0jgmUfqHB7o9zuK5zMkGIjO0iOyg== X-Received: by 2002:a63:e24a:: with SMTP id y10mr5348686pgj.176.1627709759752; Fri, 30 Jul 2021 22:35:59 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.35.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:35:59 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:11 +0530 Message-Id: <20210731053516.7700-3-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/8] lavfi/dnn_backend_tf: TFInferRequest Execution and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: zjyKX1AnfpjE This commit adds a function for execution of TFInferRequest and documentation for functions related to TFInferRequest. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 6443c2fd1d..805a0328b6 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -94,6 +94,13 @@ static void free_buffer(void *data, size_t length) av_freep(&data); } +/** + * Free the contents of TensorFlow inference request. + * It does not free the TFInferRequest instance. + * + * @param request pointer to TFInferRequest instance. + * NULL pointer is allowed. + */ static void tf_free_request(TFInferRequest *request) { if (!request) @@ -116,6 +123,12 @@ static void tf_free_request(TFInferRequest *request) } } +/** + * Create a TensorFlow inference request. All properties + * are initially unallocated and set as NULL. + * + * @return pointer to the allocated TFInferRequest instance. + */ static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); @@ -126,6 +139,38 @@ static TFInferRequest *tf_create_inference_request(void) return infer_request; } +/** + * Start synchronous inference for the TensorFlow model. + * + * @param request pointer to the TFRequestItem for inference + * @retval DNN_SUCCESS if execution is successful + * @retval DNN_ERROR if execution fails + */ +static DNNReturnType tf_start_inference(void *args) +{ + TFRequestItem *request = args; + TFInferRequest *infer_request = request->infer_request; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + TFModel *tf_model = task->model; + + if (!request) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); + return DNN_ERROR; + } + + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + return DNN_ERROR; + } + return DNN_SUCCESS; +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; From patchwork Sat Jul 31 05:35:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29149 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp135982ios; Fri, 30 Jul 2021 22:43:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzJucGA+90tiEDhul8YMOAyPr388e94pvE3vbDfHpFIiS7YG+LjbzT0SykKeIuUHMA9sBbN X-Received: by 2002:a17:907:3e0d:: with SMTP id hp13mr5933183ejc.372.1627710222005; Fri, 30 Jul 2021 22:43:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627710222; cv=none; d=google.com; s=arc-20160816; b=gWl2zFImtqPf9n8Tu3anfECNJKm8+xn3aowBZWH+LRjpI3Incrrr+p2W1bIpw7e8pS A2hMhoBwJiehgHMZxraZAZxFgznjLagR5IqRaH/+sslK3LsnSFIuQam1B0RFjDRtv8FW aQsFbmcz/041v5E93Ugnar709zB72jCE4eTiyleVLLMfUud3VG04o5u6GquH1gkfcYAd pIZ9Ye7kZJYWEzGaytbOq0keM0czcmvwqrIUwLEVeKM+chU6Bme1XvcSQWbgmYv9kOKA ndgLUyfN6D1KgvBgNKpLfPsMsKMH8DlsIHrjP2JsKvugYaH+ntuz5rSACc1y1L/dEm/4 tJCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gb/euwEosYPjiICTf+TYhcWVyXWLc2Qmqin1kXj329w=; b=h+70L4tgStS12LBHES0VyJQm6SStKe1SltWXAshQB3UnaK7v0C2ao5k49yRDnFjS9c U72Ef4ts+JzqLEvoSy3owY7D0bU528vDJqb0bqKkYtYdTVehNvSYJf3vEbilb+I5aWmn 9krCdpZCEKBlQPL4Daa/dOVSMuKry6zDYvXkWD8TGNnutxydeI02p3Tpo0wuXaYfYJVK xFbacbk5d3/dhoHOyL2mQWal2mHl+xdVDKwzEmifDiA5f3DmaTOUFRCH5zuireni9v1b QiMggxkwv+7rurr+Kw61KqUJeIUBqEaQOuWuZYtZfF+32URxVN2olUiFZMfeONeAC9jT 1d6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=pO3Y1hxf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x92si3944957edc.573.2021.07.30.22.43.41; Fri, 30 Jul 2021 22:43:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=pO3Y1hxf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BFA8768A2DC; Sat, 31 Jul 2021 08:43:39 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 62D3B68A0F9 for ; Sat, 31 Jul 2021 08:43:33 +0300 (EEST) Received: by mail-io1-f42.google.com with SMTP id o14so164779ioa.12 for ; Fri, 30 Jul 2021 22:43:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1gR2ieYZsgYdyGMlPqx2hHN0kChDeyXxsh0YdOR5kIE=; b=pO3Y1hxfMMh1YDGZW8RfNMchB1d42QnvaktSld86GR3tmuwWUn+XeChn704pbNj7X1 yXgv2CWXn/GhW19C6w65RdNWfatbSflfArqAyp8eo+Kxf1Kr45hjZtGGVS87pAIPmBbZ TRCIntYR9EUZ++Pp3xzNGXlUOzTSf2cyBOvEJ6hZIlnr+QxrBUrYc2YI9X5FiP/3VU7/ mEdOZV323OdIreH+kt72Lui+2Dy0bjHFUxSpPop6mglPJ3E24KdTNeXnP2K/KloOo+ez IodQDVfGDM7c0AOxHPnBPeEQCKzktlgAM6m7dd+kUCsYmDfPvQq9VtqxTz4I/WFiIEm/ yaTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1gR2ieYZsgYdyGMlPqx2hHN0kChDeyXxsh0YdOR5kIE=; b=t9rXXCBn+7IF6JN1l/1u9DAJiFahdygHJLBPyYk5L4+dyb6YisPvOXnKg0MZ/jzIN6 ZxO4McitlyQzm0fiStohE5ko/+doIW5rpCAXOzV3H9kpgubv+tsmyn/8QFgg2jBvtO6i 46drmNgzJnzgVyJeaAZR0qRjzg9G35c9cWuLGBVPymdZW56wILmk3wbQ4Fi7CKf6LVw+ f99pnN62CJHu3PbiYihVcCx92M92QNhsyOm5gILSUWubjPc8QXs5ZaLwdKZVpATJUEO1 glVkswuo3ek/FTCAA02Xr5BVYHG1+GTB2RaMhjW+uhn48YnDDgv381GHt4b8Jwt2LgRN MC2w== X-Gm-Message-State: AOAM533DYfq8qY9hEdcCQHNaPRwpc5QehU19Wxb/CUXRqMEw4kOYGHUF 4AcR0rtzb5QCRjShzkbWu8Lnns/busXA6g== X-Received: by 2002:a63:210b:: with SMTP id h11mr5439031pgh.6.1627709761592; Fri, 30 Jul 2021 22:36:01 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.36.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:36:01 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:12 +0530 Message-Id: <20210731053516.7700-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/8] lavfi/dnn: Async Support for TensorFlow Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: lTN//XZGYCLi This commit enables async execution in the TensorFlow backend and adds function to flush extra frames. The async execution mechanism executes the TFInferRequests on a detached thread. The following is the comparison of this mechanism with the existing sync mechanism on TensorFlow C API 2.5 GPU variant. Async Mode: 0m57.064s Sync Mode: 1m1.959s The above was performed on super resolution filter using ESPCN model. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 121 ++++++++++++++++++++++++++----- libavfilter/dnn/dnn_backend_tf.h | 3 + libavfilter/dnn/dnn_interface.c | 3 + 3 files changed, 109 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 805a0328b6..d3658c3308 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -37,7 +37,6 @@ #include "dnn_io_proc.h" #include "dnn_backend_common.h" #include "safe_queue.h" -#include "queue.h" #include typedef struct TFOptions{ @@ -58,6 +57,7 @@ typedef struct TFModel{ TF_Status *status; SafeQueue *request_queue; Queue *inference_queue; + Queue *task_queue; } TFModel; /** @@ -74,7 +74,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; - // further properties will be added later for async + DNNAsyncExecModule exec_module; } TFRequestItem; #define OFFSET(x) offsetof(TFContext, x) @@ -88,6 +88,7 @@ static const AVOption dnn_tensorflow_options[] = { AVFILTER_DEFINE_CLASS(dnn_tensorflow); static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue); +static void infer_completion_callback(void *args); static void free_buffer(void *data, size_t length) { @@ -885,6 +886,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->exec_module.start_inference = &tf_start_inference; + item->exec_module.callback = &infer_completion_callback; + item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { av_freep(&item->infer_request); @@ -898,6 +902,11 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ goto err; } + tf_model->task_queue = ff_queue_create(); + if (!tf_model->task_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -1060,7 +1069,6 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q { TFModel *tf_model; TFContext *ctx; - TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; @@ -1073,23 +1081,14 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q tf_model = task->model; ctx = &tf_model->ctx; - if (task->async) { - avpriv_report_missing_feature(ctx, "Async execution not supported"); + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { return DNN_ERROR; - } else { - if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; - } + } - infer_request = request->infer_request; - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + if (task->async) { + return ff_dnn_start_inference_async(ctx, &request->exec_module); + } else { + if (tf_start_inference(request) != DNN_SUCCESS) { return DNN_ERROR; } infer_completion_callback(request); @@ -1126,6 +1125,83 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * return execute_model_tf(request, tf_model->inference_queue); } +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TaskItem *task; + TFRequestItem *request; + + if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { + return DNN_ERROR; + } + + task = av_malloc(sizeof(*task)); + if (!task) { + av_log(ctx, AV_LOG_ERROR, "unable to alloc memory for task item.\n"); + return DNN_ERROR; + } + + if (ff_dnn_fill_task(task, exec_params, tf_model, 1, 1) != DNN_SUCCESS) { + av_freep(&task); + return DNN_ERROR; + } + + if (ff_queue_push_back(tf_model->task_queue, task) < 0) { + av_freep(&task); + av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); + return DNN_ERROR; + } + + if (extract_inference_from_task(task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + return execute_model_tf(request, tf_model->inference_queue); +} + +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) +{ + TFModel *tf_model = model->model; + return ff_dnn_get_async_result_common(tf_model->task_queue, in, out); +} + +DNNReturnType ff_dnn_flush_tf(const DNNModel *model) +{ + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TFRequestItem *request; + DNNReturnType ret; + + if (ff_queue_size(tf_model->inference_queue) == 0) { + // no pending task need to flush + return DNN_SUCCESS; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = fill_model_input_tf(tf_model, request); + if (ret != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_freep(&request->infer_request); + av_freep(&request); + } + return ret; + } + + return ff_dnn_start_inference_async(ctx, &request->exec_module); +} + void ff_dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; @@ -1134,6 +1210,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + ff_dnn_async_module_cleanup(&item->exec_module); tf_free_request(item->infer_request); av_freep(&item->infer_request); av_freep(&item); @@ -1146,6 +1223,14 @@ void ff_dnn_free_model_tf(DNNModel **model) } ff_queue_destroy(tf_model->inference_queue); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } diff --git a/libavfilter/dnn/dnn_backend_tf.h b/libavfilter/dnn/dnn_backend_tf.h index 3dfd6e4280..aec0fc2011 100644 --- a/libavfilter/dnn/dnn_backend_tf.h +++ b/libavfilter/dnn/dnn_backend_tf.h @@ -32,6 +32,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out); +DNNReturnType ff_dnn_flush_tf(const DNNModel *model); void ff_dnn_free_model_tf(DNNModel **model); diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 02e532fc1b..81af934dd5 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -48,6 +48,9 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #if (CONFIG_LIBTENSORFLOW == 1) dnn_module->load_model = &ff_dnn_load_model_tf; dnn_module->execute_model = &ff_dnn_execute_model_tf; + dnn_module->execute_model_async = &ff_dnn_execute_model_async_tf; + dnn_module->get_async_result = &ff_dnn_get_async_result_tf; + dnn_module->flush = &ff_dnn_flush_tf; dnn_module->free_model = &ff_dnn_free_model_tf; #else av_freep(&dnn_module); From patchwork Sat Jul 31 05:35:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29145 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp136307ios; Fri, 30 Jul 2021 22:44:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxDfN7FpJ4klULog/ndcuod5j7HtT5qbUZIMDwjQtgWMiYHSIULewyDAHQXzMCHgKBDedws X-Received: by 2002:a17:906:8152:: with SMTP id z18mr6308122ejw.419.1627710269069; Fri, 30 Jul 2021 22:44:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627710269; cv=none; d=google.com; s=arc-20160816; b=tEJB2HORK35fTcbwyS/Ytb8EklAJ/xjUD2nXpnyjNmpxSOcQ0sQgrEIwuIEIdGqkQu 8kltBftjlw1Aqjfz75hEDBRjAKDXrh9RInqoU1mpmFliqFy77LWYHScmDRz1y2qjT6BL aUmgvWONFMAjhUmFENzaDN7GZ/+8EPY/LaBgrl62eAqPnUYd6Fx1vY5/cppBQo+GGY3Y 8S1RHuvBJgWOFDdyAvGVrpqIaTVM+Og3LJrFSLm1fKmJBQgF4eBynv4g2wRUAWCWIZyg YvETwFTsyuNIht2ppI3E+WGNhC3U+IR9aievvoWx6j/9xP2kCWx5UX7ze8VrlI+bwABt 54rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=PquoaGuVaabw19GMBimaFa/zV1bHx/p/nPndb+VBQ9c=; b=qWMVDSHzpp4aVnQ09p0FJ2eR/LCrR5WnDg1xW3/3y5/Wj3eeifSU5Yjux1OhQoy0Eo /1y23/rugP3AI3QFOIrdN9W0uCxVy5q8x9IjK6Wm72K1sGU+WxCtipKZeYyMjo6IA8HF pP0YQeicPP/X9WTMN1m7jkzmYaCRmam80dQLWXZBD3vq0jS+unZG44u++BJltq1LRFrL nGz2jyO5Zf/9K+NUE1x2QxiJr36O4H3nUCCjxdL0qGF3dS/fytHhgkBI0XdW1hyBRO0r Gf9/55UapRZhXXoWGh9BFc3vy+YzhCpoF8xajGYMJ8HF/Uo9+jGWUr6Y9m0JORkmC9UX RR0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=bGYNZU4E; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id v6si3954259ejg.102.2021.07.30.22.44.28; Fri, 30 Jul 2021 22:44:29 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=bGYNZU4E; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CB8B768A412; Sat, 31 Jul 2021 08:44:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 067FB68022F for ; Sat, 31 Jul 2021 08:44:20 +0300 (EEST) Received: by mail-pl1-f179.google.com with SMTP id k1so13538304plt.12 for ; Fri, 30 Jul 2021 22:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vauU9bb9s1tBNjhXdaNfdF9AKP8/RU27dqP7XBh53/Q=; b=bGYNZU4Eag+F419ZuaAOeAmKZzJdPmcNx1q0s8U0enQ/+gVdtBxEFu2ev6Py+uyvjP 2cqqkb2mxRPY+vwdA/BffJLYQE0kyoMwrTMdFg6baFs1ucRJlelhCMLLgR9qCnf13YpI ilBEe1eeXWYNpnJ/paylvMX+cmxISBo28GnUrWtXpSv7JTYcqHG++NvGq9n4y/sRJCvl 7h86QnddLHNirRM+nmf7I0dgWm92BBb1TefHSLuhEhNZMl7YaccNcbcxqHLJaqUs94hM zuNm4xBfJCYJ8SU3oXD/3GS0prFOLmb26x8xol/VfGQWHTAmTvwvNkiRUhB1raf1cHU4 LnNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vauU9bb9s1tBNjhXdaNfdF9AKP8/RU27dqP7XBh53/Q=; b=ITRfK2tHvzl8Vc60JMAzTY3mjSSuVINTWpp9gLNvmS9EtGP/WDbceUiXNWIzWGelKA MqEJS9dbDdz8v50fIyo39FZIUikRZjNau3fEAtOjNCE3d1e2Cvkvwqt+L7jB75ne48uW rcpGifjuUZvbs+OQuGCJ41R2jTRp/71yL9MFE1kslVcxR9MAW6cVLw5svEGoewAcRuiM 0ETSn7yU7ften5WuH/EQclriHAwFBnaRzrByMDnbItYncCGG4C6SFEWi1lmpZf5zECBj uwXDn7KgPqYuCbVmyQZXogCoWdFdkR25eDwVPUe6kovMTTCU6IwjvA4BOYv4wq0dwLzn UYJw== X-Gm-Message-State: AOAM530zoGjjt3viTCTDEwHAkuq0AA8Iz6bpZCSeSKmzRX/9BJQcOXvs dQ9Fmk8It5m6ZfdzdY5mQTxBZhA8hVF3uQ== X-Received: by 2002:a65:564f:: with SMTP id m15mr3462108pgs.346.1627709763368; Fri, 30 Jul 2021 22:36:03 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.36.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:36:03 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:13 +0530 Message-Id: <20210731053516.7700-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/8] lavfi/dnn_backend_tf: Error Handling for execute_model_tf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OuO/OFhR3PEK This patch adds error handling for cases where the execute_model_tf fails, clears the used memory in the TFRequestItem and finally pushes it back to the request queue. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 52 ++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index d3658c3308..102e91a667 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -172,6 +172,24 @@ static DNNReturnType tf_start_inference(void *args) return DNN_SUCCESS; } +/** + * Free the TFRequestItem completely. + * + * @param arg Address of the TFInferRequest instance. + */ +static inline void destroy_request_item(TFRequestItem **arg) { + TFRequestItem *request; + if (!arg) { + return; + } + request = *arg; + tf_free_request(request->infer_request); + av_freep(&request->infer_request); + av_freep(&request->inference); + ff_dnn_async_module_cleanup(&request->exec_module); + av_freep(arg); +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; @@ -880,6 +898,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ if (!item) { goto err; } + item->inference = NULL; item->infer_request = tf_create_inference_request(); if (!item->infer_request) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for TensorFlow inference request\n"); @@ -891,8 +910,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); goto err; } } @@ -1059,8 +1077,7 @@ err: av_freep(&outputs); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); } } @@ -1072,28 +1089,35 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q InferenceItem *inference; TaskItem *task; - inference = ff_queue_peek_front(inference_queue); - if (!inference) { - av_log(NULL, AV_LOG_ERROR, "Failed to get inference item\n"); - return DNN_ERROR; + if (ff_queue_size(inference_queue) == 0) { + destroy_request_item(&request); + return DNN_SUCCESS; } + + inference = ff_queue_peek_front(inference_queue); task = inference->task; tf_model = task->model; ctx = &tf_model->ctx; if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } if (task->async) { return ff_dnn_start_inference_async(ctx, &request->exec_module); } else { if (tf_start_inference(request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } infer_completion_callback(request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } +err: + tf_free_request(request->infer_request); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + destroy_request_item(&request); + } + return DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) @@ -1193,8 +1217,7 @@ DNNReturnType ff_dnn_flush_tf(const DNNModel *model) if (ret != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); } return ret; } @@ -1210,10 +1233,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); - ff_dnn_async_module_cleanup(&item->exec_module); - tf_free_request(item->infer_request); - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); } ff_safe_queue_destroy(tf_model->request_queue); From patchwork Sat Jul 31 05:35:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29148 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp136071ios; Fri, 30 Jul 2021 22:43:53 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzeICoX7v3VP6M6cygvl+o760vuET11QUTVgtfeMBV10yAEDO9go1N6B/vxtpDz+r4RbnHS X-Received: by 2002:a17:906:8a66:: with SMTP id hy6mr5988420ejc.309.1627710233728; Fri, 30 Jul 2021 22:43:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627710233; cv=none; d=google.com; s=arc-20160816; b=hGdUEdPdhCOlk10kOpt+Dg0YCDxlf8uzv3ZfsmFnVQZUylLvOgHn6a3JjqBWqqdvkU 8Ecdv3kAU0xZ2wbZW/DB9EQjhLsKIcNaZfTiaL6kT55sgu9fxbv8YbS3Cp6Ii9xg1q5j yywTSpuoDiWpaXBYiEv39UuAplJEtrtm1HNBcW9cYnxBa2ZrzblBmhqe0VRGuzQwYIa7 ifpISTL+q43KCACNiApPXwDTogGNaeJHVQF3iYSZ4xAOES8h67QyGysF44uPjfdjvzYa TxzhAMy3P1jAH0zWQrPwjsM91WZAyyowDMW+/4uUSa6Gqaa78wlqbi9+kMMAq5uA726S oTxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=KfHwMy0bD6ebqItukOzpbVUkbLZJOnQ72zlViXBuH0g=; b=gtLkyrvsKpdrJeEBMeqSzFieBYLOtC9FyC4BABApSB5gIcdnLkOka4gFLLsQLwuCby rDWe7eieHj+1/NOlhhri9+J0jj3h8EJ14f5R4X0lkL3N6lFoyckWnrlHCOURYtN7I4Cn gRDJD5zhkCRjnu5UP1LzMzsyQsCRl2w00Aoyphk5YE2px88kIgidMM6n0jDb0ALFEOyq HAeM5U1N12Ai6r1qLFmyyFc24xjIKwu/s2X4GCTJxSHJpimOl4Xlnt4WMGFbf3Yf3GMc RIArmmM7p+kJIS3ZGwzzVuIOBc8yT6SpaUGbUApmZYu6Wx87KWXAE9QPsMJFrm3UJnAe Fg0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=adMV16tt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bq16si3760573edb.303.2021.07.30.22.43.53; Fri, 30 Jul 2021 22:43:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=adMV16tt; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC4B968A3AD; Sat, 31 Jul 2021 08:43:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ADD7E68A305 for ; Sat, 31 Jul 2021 08:43:44 +0300 (EEST) Received: by mail-pl1-f180.google.com with SMTP id i10so13579286pla.3 for ; Fri, 30 Jul 2021 22:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x6U0tprtfKfu21C6PdlWYMVrnFE50Ul36ctx/RJmXEE=; b=adMV16tt5ubk0CW//G8DwCx/CD/fsg3sCG9zLIsB15A/cDkBr1aVlOpShZi+MXklAa GbPD5HBDfE/4Lknz6fOnhJnikEMIBF7H0Wb0zVa0S+2TztP6+jtPJ9RxzSo4XbgK+Gdg /70zhfw2vXGk/g4gUm2ZDNbyIQDuhMcNi5t8xegCkRLV/rfhiNKXpB6gegY1xPxwx+R1 Nki8a023mDBzJJ1Sn47U07Idp3fXq7tHZDDMBzg5n9acCgC5vdlCjibfSLX2TI+5rT/y 5B1Bx3YI1TwVFiST49WS6HO5/+gGfpe8aKsQh9mjAP71DJEirQJPLsXZXSp3nREQVw5R eAeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x6U0tprtfKfu21C6PdlWYMVrnFE50Ul36ctx/RJmXEE=; b=AcMjYX5IuDr8gxjmoqZxi9mfbLQI/B78x2JSbyxNN0lPRmZhtq3lSTrnNoCjPvWX25 UoKM0e4XDSJ/FI02yzoGMhZzPdZ6Z15uWr1ozwjg8vCfZjNLhScpBKdn0xPvvSo2Qjai vxJUT1fEjMdiyS0YZbgvVWEMIJcE+4Noi74Zqq+o9oAlkTa6gYJAOu7oPNmrrcdkZgg8 R65LXYuIfPCNp5ENaT76fGfA8PAMBHqGHRoRnqKW4fYhH1+DzePnnVohWwdXtGNM6tvu TB8Z4c7IJNV1Z6OzTQuRXyRMdE1COUgyKTbje8yHqr4VnWvktHgw9gsPsL0fTAArLsP1 953Q== X-Gm-Message-State: AOAM531uTtoO/7Nootl3RT0De1ZhCbRgymehs+2DfXlVbpXHR2iujXT5 h4sjR4MC45qWpfMdozk53L7HhjtARFRSqw== X-Received: by 2002:a17:902:b084:b029:12b:6caa:7d9e with SMTP id p4-20020a170902b084b029012b6caa7d9emr5495967plr.57.1627709765235; Fri, 30 Jul 2021 22:36:05 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.36.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:36:04 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:14 +0530 Message-Id: <20210731053516.7700-6-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/8] lavfi/dnn_backend_tf: Add TF_Status to TFRequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: xqV8GCrTd8Bf Since requests are running in parallel, there is inconsistency in the status of the execution. To resolve it, we avoid using mutex as it would result in single TF_Session running at a time. So add TF_Status to the TFRequestItem Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 102e91a667..c3425e01a8 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -74,6 +74,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; + TF_Status *status; DNNAsyncExecModule exec_module; } TFRequestItem; @@ -164,9 +165,9 @@ static DNNReturnType tf_start_inference(void *args) infer_request->tf_input, &infer_request->input_tensor, 1, infer_request->tf_outputs, infer_request->output_tensors, task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + request->status); + if (TF_GetCode(request->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); return DNN_ERROR; } return DNN_SUCCESS; @@ -186,6 +187,7 @@ static inline void destroy_request_item(TFRequestItem **arg) { tf_free_request(request->infer_request); av_freep(&request->infer_request); av_freep(&request->inference); + TF_DeleteStatus(request->status); ff_dnn_async_module_cleanup(&request->exec_module); av_freep(arg); } @@ -905,6 +907,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->status = TF_NewStatus(); item->exec_module.start_inference = &tf_start_inference; item->exec_module.callback = &infer_completion_callback; item->exec_module.args = item; From patchwork Sat Jul 31 05:35:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29144 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp132895ios; Fri, 30 Jul 2021 22:36:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyLgND+p0zaumOLsYPXWAX4cgFS/d5/iNCLfFyMKFg095046VWIH+5DkbLq3ciEX03vIlV2 X-Received: by 2002:a05:6402:22b0:: with SMTP id cx16mr7397915edb.185.1627709780299; Fri, 30 Jul 2021 22:36:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627709780; cv=none; d=google.com; s=arc-20160816; b=MxrfJjFKDy2QdKBSfmej58NVwCNvoMTApTHXQ/LntAAiDL6SI5gBQG/ihtofgyVJ0/ 3TMsG7fLbH2okeypLUaZe2F5E1LLnlk69b1FC3HCFEH0fUyr6XWqSybPZ8VEsYDVpkmi fZcZ8VMz40nJlOGW/uUhMLdiqJsCuQXNq45JEs/oxm8EP7vKo7u+ccnRZGfjLRbviaiK nN4j69In3f9OkM/RfTdwDZFqgrujGxBJTn6R3bp2VIuncQiSGjuwf81pJU3kYI5zc4Av E1uONvXoAqwqkueF452qQneJM6pvugMbsEWWW8XLH4bPYslPSj0dz88tZz9XJKOkKnJ/ f6fQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gRyICjugDiayJLj2iEvxw20o6U4kNhcX198/MHGBwuo=; b=VMhdhPvlLhripJkIUtctH5NjrGiBWs7OYLteX1+5x5ILjTkDHbIoMTRrRfeteiCcAz 7uB/h6lgoj5S84cHyfuR/Bci7Y/RJm6ZcBuV06gNekQz9lob8g+3p8JeJdBn+FySs4zu MkiaS0vQl1BGQXW13WKyyBKZzOGVLE7y0+uTwpvn4gIAtWiY5y+zI8nwmjjrIZ9mTL6E XrwUuFHryguwzyjulD85giFGScenrYcD7NmICSUDBQt7IEnPuuFnRUprgfwi7iTJ9mOX OQOXgruHiECQmjB2XrKKaEwusNA4pStASpeO0BN9PD/NRmsKDfnpGnSIDy1eh8zARTbG QHWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=o1kTUNcc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id lv9si3887292ejb.470.2021.07.30.22.36.19; Fri, 30 Jul 2021 22:36:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=o1kTUNcc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B53F3689B83; Sat, 31 Jul 2021 08:36:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E6C5A689F02 for ; Sat, 31 Jul 2021 08:36:08 +0300 (EEST) Received: by mail-pl1-f169.google.com with SMTP id e21so13557422pla.5 for ; Fri, 30 Jul 2021 22:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D4wOKHSoELtJQgdBzUqXxpwkADqTvb2S78r/abmBbV4=; b=o1kTUNccN8GyPgj/GPS6gu/akMkYPFxKVpA7BXs8Cbtz6fqU7ozjF9uX/H1KUVpFtN IEDi7nbnjh4uWstoEwEJp0au8lDFITVkjG9DqhMha2m+VegCh/+0h9IEYhOLzcyrgCzF CrVWb8pGng8qws5GQIFe7TmgEzh9eF1rDooHo8zX/oqp+XVim4nlWeeSwMfde9ibICAZ XUFdDGhWSpMWbe+MsdyMGXQHHDANYgM3PhpTyMBopDj68JipHMOlFr9WbhpDitNQ2H5A A8AZS/D7GVPGmzSSA43pRTdOHc9PjQ6ZdeCAoUEnB21ctFISHRdBwGnn62keg+PnmJT1 VIWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D4wOKHSoELtJQgdBzUqXxpwkADqTvb2S78r/abmBbV4=; b=pWRWaZG4dZH4xY0UBhMdj/40nIDn54t0pxirD6qDt2zcRfHpGO/K8okOYlfOOGG/g5 3I8w7m8Gu02yxB/psjJ1tiVP08rPQ5BxhLQDP6Hvt5wFlSODg99GzOGHoru/T5/Wwpge LBGiVW/US4W4km+zGrSX1nU2i8tSGvdYT6NTlOklalHnHfTvBJRBcpZqVXyfdXHNjIEj 7ncLRInH8FbpB8+mmIxTwK8zdJ8egub78lWvCZsWswTlZliD8dB5p5elR/dNPWNn9x9f /03/AgAuDfcTMZd+cb55q0w3A9Q5T2+9OfqMnx/++8YFgOPohEsZe+zQNs21FxD8ezRL uxeA== X-Gm-Message-State: AOAM531CEEX6g7ryb7msue7iiemGLm+mRpIlVJH2eu886i5kOYFWJ/Xx FklIGccOHZ2SHI7A1a3b0X6MCU0WQ1PlWw== X-Received: by 2002:a05:6a00:d5f:b029:32c:7264:2f65 with SMTP id n31-20020a056a000d5fb029032c72642f65mr6115128pfv.40.1627709767067; Fri, 30 Jul 2021 22:36:07 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.36.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:36:06 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:15 +0530 Message-Id: <20210731053516.7700-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 7/8] lavfi/dnn: Extract Common Parts from get_output functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +R1hvivr8MTL The frame allocation and filling the TaskItem with execution parameters is common in the three backends. This commit shifts this logic to dnn_backend_common. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 26 +++++++++++++ libavfilter/dnn/dnn_backend_common.h | 16 ++++++++ libavfilter/dnn/dnn_backend_openvino.c | 51 +++++++++----------------- libavfilter/dnn/dnn_backend_tf.c | 42 +++++++-------------- 4 files changed, 72 insertions(+), 63 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index df4bab85e0..470fffa2ae 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -142,3 +142,29 @@ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **i return DAST_SUCCESS; } + +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx) +{ + AVFrame *in_frame = NULL; + AVFrame *out_frame = NULL; + + in_frame = av_frame_alloc(); + if (!in_frame) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + return DNN_ERROR; + } + + out_frame = av_frame_alloc(); + if (!out_frame) { + av_frame_free(&in_frame); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); + return DNN_ERROR; + } + + in_frame->width = input_width; + in_frame->height = input_height; + exec_params->in_frame = in_frame; + exec_params->out_frame = out_frame; + + return ff_dnn_fill_task(task, exec_params, backend_model, 0, 0); +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index e1d678b230..604c1d3bd7 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -137,4 +137,20 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ */ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); +/** + * Allocate input and output frames and fill the Task + * with execution parameters. + * + * @param task pointer to the allocated task + * @param exec_params pointer to execution parameters + * @param backend_model void pointer to the backend model + * @param input_height height of input frame + * @param input_width width of input frame + * @param ctx pointer to the backend context + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if allocation fails + */ +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 47cbd48ba2..c825e70c82 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -640,10 +640,15 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu OVContext *ctx = &ov_model->ctx; TaskItem task; OVRequestItem *request; - AVFrame *in_frame = NULL; - AVFrame *out_frame = NULL; IEStatusCode status; input_shapes_t input_shapes; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; if (ov_model->model->func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); @@ -669,51 +674,29 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu } } - in_frame = av_frame_alloc(); - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, ov_model, input_height, input_width, ctx) != DNN_SUCCESS) { return DNN_ERROR; } - in_frame->width = input_width; - in_frame->height = input_height; - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; - } - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.nb_output = 1; - task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } ret = execute_model_ov(request, ov_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; - - av_frame_free(&out_frame); - av_frame_free(&in_frame); + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; +err: + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index c3425e01a8..9c96868fab 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -309,36 +309,20 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu DNNReturnType ret; TFModel *tf_model = model; TFContext *ctx = &tf_model->ctx; - AVFrame *in_frame = av_frame_alloc(); - AVFrame *out_frame = NULL; TaskItem task; TFRequestItem *request; - - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - ret = DNN_ERROR; - goto err; - } - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - ret = DNN_ERROR; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; + + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, tf_model, input_height, input_width, ctx) != DNN_SUCCESS) { goto err; } - in_frame->width = input_width; - in_frame->height = input_height; - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.model = tf_model; - task.nb_output = 1; - if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); ret = DNN_ERROR; @@ -353,12 +337,12 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu } ret = execute_model_tf(request, tf_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; err: - av_frame_free(&out_frame); - av_frame_free(&in_frame); + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } From patchwork Sat Jul 31 05:35:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29147 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp135668ios; Fri, 30 Jul 2021 22:42:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyEP5y47gVqvVmz93t1S8irkYTsEIfMoEKH3RmV0V0YWbTwSC92ZBHnodds+B0tZ2iwbxGF X-Received: by 2002:a17:906:2813:: with SMTP id r19mr6029677ejc.150.1627710175342; Fri, 30 Jul 2021 22:42:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627710175; cv=none; d=google.com; s=arc-20160816; b=vg8J669LDtoZh3J85d+RLVmF3HrbTbQQ+PY36H83lAV5LflVF0850an1cTIsFviZ5N bSqu6jeVwXlMc+jx8odqRz1agpTp/D4gpLttKyFmjVa2+k6SibHoeria9hLcEl9Tg+8k dML8m1BEbSDQNHp2K2ywfW67pV7Y3zTHSHJKGG6hrFC7IbQWqC7icHgHYMP4zVlZXk9W M5tfIWG+0GsVYf9EH5LjHau1Tb1ShZjJG66A2KzKIlwUvFQbYDU39ESwntalQ+5otGQE wp4dT5UXDZzfVMK25+a03M0qNbWTCQky3XJSAKgU8bu5xGZsCKArY19vwC41K3GcLGmd 8Wjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=N8W2OlMokVJ5dfn/FDWicEHHABQboJ118xfqHg1SyW0=; b=EP7DF/D9GqA5lrVfHPN2IvAKLrKMYxvb85vUjj8QaeBL8DWSZMjed3G2fOYZQ1EPlo Wu8j2GYbuDuKqK+kRK9wgVLcD5VW3o31j9qXSuNUyEekRl+5JYR7AjGXZAciP/K2smOm +skhL0bYIfvuPrL/FcuWEk+3VP/eY68IP1zvFpQoEFL5Mdl3vTyia/qSNWwExyw8bDxR wcEzLAk9yKvRwrP5xYNuNRWpz3T0uaHklBeN3nW1EeY9s2vzkOeXugdjq0gigqtoW/vP fg/W4YGQh4b6UAamHCJjOCeKtaBoIn31Mo6qskAgxiB9RYvUYAstkZf3HcBu3ZnTECjK uuVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=V+nssGHc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hp8si4310000ejc.49.2021.07.30.22.42.55; Fri, 30 Jul 2021 22:42:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=V+nssGHc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 83D9F68A2F6; Sat, 31 Jul 2021 08:42:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 94E0A68A0F9 for ; Sat, 31 Jul 2021 08:42:45 +0300 (EEST) Received: by mail-oi1-f170.google.com with SMTP id t14so16433712oiw.0 for ; Fri, 30 Jul 2021 22:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BpRvmxEY/uKnNpz71U+knN1cWsUaMH6abM3kZadvY6I=; b=V+nssGHcvTgIcW6Gu33P9uuv5Gp1y9rLhrYqxwNUDWxbtuHBWpj3oFNkklWHEAl1tC jb5AvcNVu8u+/wx+rbDhPxUrhMt+yo5Oj9J6vsIkGcrluWhNQamofoO/zS61TH1nwkth wRGXbLGITGorykTExQ+EpFt7fZp0Lq8OlGM9K+bwYUoTIevByDW7NlhyB930WZgwBlZi KIiE8K+HtzFvZYklOrh9G/lfFiIFf0AQsPoPcd15MorQqx2UNvVC6mw9FgaBjcgzl0w5 36fpgOPEgcZRzmBe5Y8w2ZQEKXCgyGUB3+gKOJHv3UjcLDjKHN7+ctIoCwMJKyELn56X 0Kow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BpRvmxEY/uKnNpz71U+knN1cWsUaMH6abM3kZadvY6I=; b=e4QPpdaqbkSyzpXhSjB0jcfUKhSD4czUxX9ro/aS7aZq0VZCGsat8AeYIbs9PhH4N+ T27h+Iqyz/b+gn5i93KVvg+J94ZneES2cIgUkNNnreIS/8zdSn7L2KemPGgiaRTBCocf BtgKsm8jd5yxPLTOfOatQXmknWHimr9DP6u8z8R5mETe5qMfrpb51X+H5UBaONjm04Vh wCZNg6uHR+J3qP1PlMZl6m5uSXeD4OZ6gFWof6TbiMbWWoVghd4UkF7V4JuJEO9NiP0S 18Z3SMbQBWRlx9jCydN9dmC8ou0kM8HN4etXvouQaX5cA1xM6a6l8zKGNAC39mtBrxU2 hxVQ== X-Gm-Message-State: AOAM531lXbU4GOurO3fFuN9oOJLbwGPjYuqcyjzl2Hsm002t18vWfuNI A8K3GEsyXMCMBlGXuwT0mR7zxAyra5uz9A== X-Received: by 2002:a17:90a:cf94:: with SMTP id i20mr6544966pju.219.1627709768774; Fri, 30 Jul 2021 22:36:08 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.36.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:36:08 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:16 +0530 Message-Id: <20210731053516.7700-8-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210731053516.7700-1-shubhanshu.e01@gmail.com> References: <20210731053516.7700-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 8/8] lavfi/dnn_backend_tf: Error Handling for tf_create_inference_request X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nJs2cETYNvm6 This commit includes the check for the case when the newly created TFInferRequest is NULL. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 9c96868fab..b14b621bee 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -134,6 +134,9 @@ static void tf_free_request(TFInferRequest *request) static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); + if (!infer_request) { + return NULL; + } infer_request->tf_outputs = NULL; infer_request->tf_input = NULL; infer_request->input_tensor = NULL;