From patchwork Wed Aug 4 11:51:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29233 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3307106ioh; Wed, 4 Aug 2021 04:57:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJypVMM+/0+qyCL76WloHUJJsryEuZAk9UHDDxZkE9P3QBbwEfFErrV1zgR3EEa4AG6clu7C X-Received: by 2002:a17:907:d09:: with SMTP id gn9mr25712009ejc.447.1628078253831; Wed, 04 Aug 2021 04:57:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078253; cv=none; d=google.com; s=arc-20160816; b=v0mFrGQ8eF4TTnsA/wdM8kKcY+RkWmBrn9dXXd4HqVVpEokEoWqXp2iwPJ2gZnkwQp 3EWgGZHyDFeUvRmw5ayBlI9bsd9ECcoodZTlW4Ro7ebT/e74JT+ObU5o/iqpsAFF2ssM MMT/716bdr4i0CrcVGoNdgZ5xsMFmLA2XTs4TzhNfhMnU3yip47pSogwN9ywXBf7Zw5P yoZhkbNwxCUxOMzg12h3/pHyXnNOJOoPV9YQLkuMhq4DNBa3VnrjS6pX8qbY8x0MCXhK 6ijbScnTK0u14ltGu9zI9XSV1Q3p8b4ndcWnnMeTW6CLfAUvphZ3syIdLXeKE1J9cQ3I 6UKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=eBVHHWZ1lgEttxcEiYZxSDvSzaC5iqBfqr8PEh6kF0DhTYTpp3dPnirhk1Ju2kdzkb d81npa/lOINpkpYlo114368z1qYghVqGbvCiC7sL39cspx2gDWGhTs0nOgbWOOp6XsTN WCeZv4FoYZ43RZdDUT7Hqs7PvKfM5ITM4oL1lBMdwzl7QbL3jnFa9FixFVAbo5GRiaa/ cJk1GQghhDE49LCAJaoOuYPdl4CEsvRtbVEqi4wbv6NNBNd+EiAgzSbzZYBumN2Zx6H4 KiJrsxWHkqjdtY1CiyMeZKFefl0L2dbK5AYNkAnzTYD9jHorNx5tqJOKKGcgogFR6fKF PaQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KyPUNnn6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bj22si1976513ejb.53.2021.08.04.04.57.33; Wed, 04 Aug 2021 04:57:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KyPUNnn6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8F3EE689B19; Wed, 4 Aug 2021 14:57:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7F18B689B19 for ; Wed, 4 Aug 2021 14:57:23 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id l19so2639440pjz.0 for ; Wed, 04 Aug 2021 04:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=KyPUNnn6frBA3Fb5o8YYa+8M2N2QAFXeWKlMCUkB+tvqKfg2zm3YMNCob0Qxpg+SJ2 h0lWWdOvb2thVd/nTFADZzBLbMfLl+SWI/xbcXIPx4GENVeQ+sXEHvwhWvodZulMfZNy 0KXXbbFD2S8gndqSxm09LT8DLZbj0m40BCpS0KfZruEkMTgM9z3LRhAxn5LovoH9DyVd HlCblIeS59JASqmy2U0Rla+ulle4BlNi1GYhx2aiSCGLvwNFisCojju1sqIVo7tGSIMU 3jltpTm/X8Dmss+HJXY/i0YPwcCQy4cXRkoO4/cgtzgDdO42cmnSxfbka6zQfyv6qCA0 Rang== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=prc5cJT5SU0MmmNUUBHM0XYPrR5o1Ce7YQZgCyggEY9mHF67G5Sf5vCruA/wU+GzJk Dz0+53fFCvAB3+KByl3iscGviv8xRW8XSk19mLv1Jn0QFQknx9p2RvmSPKDJVtUEwH1o fY+DrLp5kX02XO4KowncojGqefwcY0z5s9OpcoADcptRXsiAXr1qxqiE2yYgAfVeNC2y ddkl1lLR0nemhquUvMyEYFDRz+pMRpNTnk3kKPXuHJ3Dl54QnSyU6GT/VWeLbscb6jc8 Od2SU3+aANSQ/7Wq2srB7Ic2PDA2uC+91nyhUUtwut2Q0SfDQL49G43iupu6keFBFKwS 1EhA== X-Gm-Message-State: AOAM530OYspWE32/pW3ve4kC6Rp4vaXyLGNL95j7WhZcunagGlupJ9kD QqPpbV2QVzK5WdGDp1Wa7AboPY6LyErLQQ== X-Received: by 2002:a17:903:248f:b029:128:d5ea:18a7 with SMTP id p15-20020a170903248fb0290128d5ea18a7mr22573843plw.83.1628077919897; Wed, 04 Aug 2021 04:51:59 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.51.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:51:59 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:31 +0530 Message-Id: <20210804115138.64475-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 1/8] [GSoC] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZeJwBXbYFZHd This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif From patchwork Wed Aug 4 11:51:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29242 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3356414ioh; Wed, 4 Aug 2021 06:04:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKsS7XwRnVnWZkEsYW3xlH0xvYP5smEOuZZpNCwdOeDjNlmY9fGSGL6BOurCJh5ybpZFwx X-Received: by 2002:aa7:d748:: with SMTP id a8mr31670763eds.169.1628082241967; Wed, 04 Aug 2021 06:04:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628082241; cv=none; d=google.com; s=arc-20160816; b=T5lXz+PAUJNDNcsOeH3oAvRspfgYds26sLts518hwvVj/GhuMMK39tOntG0wta15LP ++ACBATphzs5wv6BiXLhwIHehjVjpdzjHYUDZEbEx4tJAaaFfPm3byloJIPbi97dgdEx r70HpU0V+Alsyne+rHed77OQUsQNK9WXpRbnkajcWFAJuiJdZNgchI47eXrD+fxKHyD1 cj3RI6BmwonZyowXnfLyB8hMgFk2f5/8qH5oP/bhyP+23ZXhgy8DSyHobhAvjsxbvTdA EKrwZYZYk46rf66LZeakjMHusdjd0ra1ZrgJY/2aXpYitgBy0tmki+IDuFxi3JmCajFC G5lQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=2BaZt/nZ8tYkB2dMxu8/Sj+l4fN6MVIaca0p0tqv2lg=; b=ixqbyMydhAyhJhiHLCI3rL13RLuGsGJngD9GfGZwBfPdJxDVonlQYpnbG4XDHVnaG6 +wKwSlxv+Cq+EfOtoLrakeaIB0X0+zYBjPPrJyXOKNCI9Eu0aJGBnWnlj3AjY2sCHPou b75A7/2gLzfb/6JVcVwSPBswWvKi5OZpLH3ci3mwLjXU4XGrQNqWzaUZITCHXYaoWSLa bv3Cf7TBNvpAXI8ejAuyW3HZXvJKoQqG5bul6UhGKKCG6u1taZoNHuKTn2douPl9oQrB aY9J5ZmNT4oJHdhWMv7BPLZj9hBWM14vb5IkXM8roLFG4OtGJi/H8xR4LlU3elJPo0Js 4qKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WNnz6LCf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y8si2059112edu.495.2021.08.04.06.03.53; Wed, 04 Aug 2021 06:04:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WNnz6LCf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CFCF068A073; Wed, 4 Aug 2021 16:03:50 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id D0E1F688178 for ; Wed, 4 Aug 2021 16:03:43 +0300 (EEST) Received: by mail-pj1-f44.google.com with SMTP id j1so2835343pjv.3 for ; Wed, 04 Aug 2021 06:03:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=WNnz6LCfTcZT9zCL39EwYRK4lXpfzX7rL4Nxs2aMj3EW6psHSJ774477rbLF4BtZkE L3pQQJvjV7z1gwI5/vjCfE42yPvb/+vMOauUIvrG2uEYWLb8ClgysdobLVnaDPx7tizi fA+vBf18wRkdRcBz5Ovs3eEi37G5Woq6RcF7JGCT9RoAzi9wx/5R7lvJVV7fvf+FGQkN yvNcQshtz7ePMgocs/mO7GqRO2VGsakVn5TfLvgeSlEnxf+wX/9edXqMMfNWiXYNXr2c yKMDu213cTy8pkB+7v3fZPdJzBIn9JCGAk6WICmpnZDnrZDKjsUu59v61UCAsSr8nzKc nBkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9+fJlnI+zvoCClE87bRDOXCrcDAq6TMyrQhV785zezY=; b=oBP4LtaOg0uMEqj9CUEvN+LTeIKme4LnMjixo668EJZgllgugBp1NBqqWPYwzxv4Qz dQBuz7uDYml+4FDbJnpjhjdUT/z7wf9GDXzQkPLb+lwqYBl4Ux0Lhh/0Vvr00TahcUsU uFPRCQEK0k/fGMDMmsNtSwJETqYaARBdH0w7v3Yw9qd5a563HpXgldLMSE0iKG16l4U+ MC/YrinKUawutLsoQUVLnKwriCJlcq+jGjZt3tNl87DHOSA8eTwOYgG0GOqfOZS8dOqB EBCXYq4RVEWPU6nQhLxUR8b6Jv0Ar604stB6W9DOj9mDeVHibl7eT6p72CJC3QnR3F38 4+2w== X-Gm-Message-State: AOAM530gfAnDfowO0yK8LVy6fsLS+VrBDkoXbJSDACxVAl44aVjZLu7M g+kimnbITMaBhoDnUMkZ46eG3QKn1bDRXQ== X-Received: by 2002:a05:6a00:26e5:b029:330:be3:cacd with SMTP id p37-20020a056a0026e5b02903300be3cacdmr27165820pfw.78.1628077921903; Wed, 04 Aug 2021 04:52:01 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:01 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:32 +0530 Message-Id: <20210804115138.64475-2-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 2/8] [GSoC] lavfi/dnn: Common Function to Get Async Result in DNN Backends X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BnaAoHViIW4E This commits refactors the get async result function for common use in all three backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 20 ++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++ libavfilter/dnn/dnn_backend_openvino.c | 18 +----------------- libavfilter/dnn/queue.h | 1 + 4 files changed, 37 insertions(+), 17 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4c9045501f..df4bab85e0 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -122,3 +122,23 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ #endif return DNN_SUCCESS; } + +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out) +{ + TaskItem *task = ff_queue_peek_front(task_queue); + + if (!task) { + return DAST_EMPTY_QUEUE; + } + + if (task->inference_done != task->inference_todo) { + return DAST_NOT_READY; + } + + *in = task->in_frame; + *out = task->out_frame; + ff_queue_pop_front(task_queue); + av_freep(&task); + + return DAST_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 96e4df9676..e1d678b230 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -24,6 +24,7 @@ #ifndef AVFILTER_DNN_DNN_BACKEND_COMMON_H #define AVFILTER_DNN_DNN_BACKEND_COMMON_H +#include "queue.h" #include "../dnn_interface.h" #include "libavutil/thread.h" @@ -122,4 +123,18 @@ DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); */ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); +/** + * Extract input and output frame from the Task Queue after + * asynchronous inference. + * + * @param task_queue pointer to the task queue of the backend + * @param in double pointer to the input frame + * @param out double pointer to the output frame + * + * @retval DAST_EMPTY_QUEUE if task queue is empty + * @retval DAST_NOT_READY if inference not completed yet. + * @retval DAST_SUCCESS if result successfully extracted + */ +DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 8d23446296..47cbd48ba2 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -32,7 +32,6 @@ #include "libavutil/avstring.h" #include "libavutil/detection_bbox.h" #include "../internal.h" -#include "queue.h" #include "safe_queue.h" #include #include "dnn_backend_common.h" @@ -883,22 +882,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa DNNAsyncStatusType ff_dnn_get_async_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) { OVModel *ov_model = model->model; - TaskItem *task = ff_queue_peek_front(ov_model->task_queue); - - if (!task) { - return DAST_EMPTY_QUEUE; - } - - if (task->inference_done != task->inference_todo) { - return DAST_NOT_READY; - } - - *in = task->in_frame; - *out = task->out_frame; - ff_queue_pop_front(ov_model->task_queue); - av_freep(&task); - - return DAST_SUCCESS; + return ff_dnn_get_async_result_common(ov_model->task_queue, in, out); } DNNReturnType ff_dnn_flush_ov(const DNNModel *model) diff --git a/libavfilter/dnn/queue.h b/libavfilter/dnn/queue.h index 2524d5fa59..9ab5c89f65 100644 --- a/libavfilter/dnn/queue.h +++ b/libavfilter/dnn/queue.h @@ -18,6 +18,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include #ifndef AVFILTER_DNN_QUEUE_H #define AVFILTER_DNN_QUEUE_H From patchwork Wed Aug 4 11:51:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29237 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3308525ioh; Wed, 4 Aug 2021 05:00:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwKh3x5jmVg2v1krCj07w6jwVwTljb2Xr6gFTCeYFTmbaTdAS/VQQO7tq2R94DMsP3IVOKw X-Received: by 2002:a17:906:6d85:: with SMTP id h5mr25316697ejt.305.1628078402817; Wed, 04 Aug 2021 05:00:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078402; cv=none; d=google.com; s=arc-20160816; b=E53AynjPQZ0PnUwc9A1/WkkXRUY+M/q8JSPX+hb0LpkF6jg9YYsN4Mpok/jHdjCBHV dZi5rUi2PKNz2wuT7mCZ2Wt+oN5BQBv3xJyG8t8rOZIvVnfXZ5FMsRY4L0aLSKv0pReH c63tOWG2BkvLrcwY1sReLr4zmfw5EfWwSUS1GLHNH4vhgf+YSpNGz94mLSWw4y9Me1k3 nybHCv6ZtRKdWvUIMpj/RlM2gMToWUlxWe+M6ow79y0mAAlTS11IYMQDN23v4psZ7q+i xdEPstzBZQavglfvBb7HR3SNpojN7W1PjGKxNttztj76ufeq0TUr/qYIS5V6xWA5FL1M +J8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gZq5amk4Fenj6Z1yPnJLFOm0T+gDdQEBhnvRZMDobOw=; b=MhLug88po12P7ENiBMm1qXETwk7PiRKAMYaU0SZ7bWUpLGFhM/Qa9rUh306FBXrAD8 3qV/k1jxDU3uzhAaGpN+KoOx7UN97r/DrK9FE6uRumr6cNpiLcpbMPU+tPZJ5MxYP+HT XU3DM8S+CaYiMPLGw6dYcoem3WuT5RLyFz5G7QJwgisVEKBPzsD5vtLgQOEH7Rs/xWVd yRO+WXgCmSh2Lq5XG0bQXZGh8u3Izi1jlkAGgn8aFMhgqOszTAlEkSAABTlj8Zwfsc6F 5QbcIBxeO7CFkEzrSH6XoKQ9JREs62Mjps06rUvordsg36Qp4xZziNIloymmHRNZxgbs JTBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="ZxwD/OPh"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u61si2216716edc.99.2021.08.04.05.00.02; Wed, 04 Aug 2021 05:00:02 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="ZxwD/OPh"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6384668A073; Wed, 4 Aug 2021 15:00:00 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A3D0C689F3A for ; Wed, 4 Aug 2021 14:59:58 +0300 (EEST) Received: by mail-pl1-f178.google.com with SMTP id d1so2755186pll.1 for ; Wed, 04 Aug 2021 04:59:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cmzj2KvWrvFM25vmhuO/8auwNOwo3QxtZvI7+o3Mr1g=; b=ZxwD/OPhz2DwcOsHUGJ38qTFGrebqtB716d+IXjZw311uOXVKKmfd3NMs/r7UUAx1x HT8+334CDqmHIMnKHAw3ezHl9J8SiBMYsjbQiG4SzzXWqaJAGzcoNwBxHQUP3seW1TfA IKHGPSkubwhKHhDPKIBmfLXaN6k6nFTOI0uwMVDmBJ35hqey7LVmAt6Ay3WDGe4ToS+d h1O/cplDZ2rRvS+Z6GHTsMlT/JGnahX/c1InYkUc55ZjrNXX+CVopcAoJK30+Dex23F7 98g8dw+IkprBDyjPCmQoy2/Dtu7AEtDQtwn4/10o75ovhSboCNc2y2pZByPpq8InePN4 vO9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cmzj2KvWrvFM25vmhuO/8auwNOwo3QxtZvI7+o3Mr1g=; b=PeAQSuMSo+QItalrCs/V9OZkbzCfyd4s14GwBwAhd9mAYlB0kMbTYDl8NDFSxXb4Zl Sy9+H1FYFEU7IBjaNfrZSqW/y0dtk2S0VaXLUxl6EfEVSBo54GbHW5p82Bjjj0kfEqto K6PeNoHcwLGOBZdcnDMLmh+Jrgozo9ibaHlrY+8BOIN0NU0m4b122D/W9rHLTgrw/TTV eD+r5T6oVCpEfKWD2b4MX2NQG4urUtFSu8OmQo+3R4WgjqAnciCwMlOuIY7DSpl0er76 S9zUrULqccXIyeNpPnlEoU7p0DK8FbYWVV5mQVIKS9/6NwNQKpqL0WTzr3At3tZqK/ut 6QZg== X-Gm-Message-State: AOAM532k1U4JdJHydRKGJps+X0ipFQmhTXpE3U/Vsea0sS0i8lgwr+FX Mu/AQPXxNRgkjlyi2tIJEznkTdp586cZXw== X-Received: by 2002:a62:1892:0:b029:349:1b8f:af2b with SMTP id 140-20020a6218920000b02903491b8faf2bmr27644070pfy.20.1628077923750; Wed, 04 Aug 2021 04:52:03 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:03 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:33 +0530 Message-Id: <20210804115138.64475-3-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 3/8] [GSoC] lavfi/dnn_backend_tf: TFInferRequest Execution and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: BgbfURSkAZhP This commit adds a function for execution of TFInferRequest and documentation for functions related to TFInferRequest. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 6443c2fd1d..805a0328b6 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -94,6 +94,13 @@ static void free_buffer(void *data, size_t length) av_freep(&data); } +/** + * Free the contents of TensorFlow inference request. + * It does not free the TFInferRequest instance. + * + * @param request pointer to TFInferRequest instance. + * NULL pointer is allowed. + */ static void tf_free_request(TFInferRequest *request) { if (!request) @@ -116,6 +123,12 @@ static void tf_free_request(TFInferRequest *request) } } +/** + * Create a TensorFlow inference request. All properties + * are initially unallocated and set as NULL. + * + * @return pointer to the allocated TFInferRequest instance. + */ static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); @@ -126,6 +139,38 @@ static TFInferRequest *tf_create_inference_request(void) return infer_request; } +/** + * Start synchronous inference for the TensorFlow model. + * + * @param request pointer to the TFRequestItem for inference + * @retval DNN_SUCCESS if execution is successful + * @retval DNN_ERROR if execution fails + */ +static DNNReturnType tf_start_inference(void *args) +{ + TFRequestItem *request = args; + TFInferRequest *infer_request = request->infer_request; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + TFModel *tf_model = task->model; + + if (!request) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "TFRequestItem is NULL\n"); + return DNN_ERROR; + } + + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + return DNN_ERROR; + } + return DNN_SUCCESS; +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; From patchwork Wed Aug 4 11:51:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29240 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3326669ioh; Wed, 4 Aug 2021 05:22:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyNezEMQpE/8JDECyxFbR41B9LB1XuGrmf8rKvkzZ7NT+of9Cg/MJoyxu914aVgGTiv+wx8 X-Received: by 2002:a17:906:705:: with SMTP id y5mr24877914ejb.149.1628079776556; Wed, 04 Aug 2021 05:22:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628079776; cv=none; d=google.com; s=arc-20160816; b=juJBih7ooTkqINKkC9GVctZmYpU2GMhsBEXbjXFvuOyoMagA36B3RUEGjNodd+mMBk ZdPLZxOaeQclfBInYtOOsLdZPRt0EnRWUNLnXj97Vd/rpoX9kq/OSG4YPnelRXVmKTTa 44ssk6Q6k6+83GvOdSizN44uzhibK6svtvbz7TC8FqXGVZ0PCz0Hjq+xT0GAsxLSrC2v Y/EBR4WKzdDYavKeZZEb4XOJ3IKW2kTN6br1axILCpRfHkIYpqdtPDnfcZk3DMWBVQUn 2o0U8Bm5hGM8Km9g2qHw9g7DX0stHozXCVDuEcVBz/GnSis5g5wnE4OUgBQ/1AKXraqP UPzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=V1+ICQ21S2gPq0yCfj6r+YiDN7xWoN8JN+sdt1HlhDA=; b=RpPUcgb8+xa0z9a9uHGffzCIWb2a0oep1Pf9K1Pgi3eOkQIjntTVSmJmJPvSzVtYXg WBHEeumnP1A1Ucey19XpFmSdXwQEs1TGntjVa5lcxXmJcd/6h6c7JM6I3HNRKUGMbcs2 ZArY1vv2QS+IJ7WAyQBEMjKc+R8YHzRYa9JGf5dpuP3Kvi1o0CqqLigEEIKRpLqQEBFX cut5h0BpNP/YsxoA1S6JSCsmU/1DmViTfj4mGH4HWuE60Rda7e+hh8FGLe4iBOe2Bzc5 wf081x0CQ3dvdnaFQpRMbbAI776VFBkplAZUnq4keBS03k2KP2CZw9m8MisLyw+7DMWn XwFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=UAgxzBIw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id c8si2055092ede.304.2021.08.04.05.22.55; Wed, 04 Aug 2021 05:22:56 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=UAgxzBIw; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5A0B1689955; Wed, 4 Aug 2021 15:22:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 74E4068806D for ; Wed, 4 Aug 2021 15:22:45 +0300 (EEST) Received: by mail-qv1-f54.google.com with SMTP id kl2so955124qvb.11 for ; Wed, 04 Aug 2021 05:22:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VmvynFoDx+JpYeqIScffxYLG+oVJVOqfxROPxE6S540=; b=UAgxzBIwH5laIRxT2UoYsgPSTXR7fxedsTOZnnp+Rt0u+z9bpwJaWJ88QBT+OJ2FNi tVbVvkuPY+YyP1Er8e6Af9p2ZJlPzgHQo757EmKfTF0IPch63527/quhW5gq0sCJk3UH amjNc6YKVOeaw0CTQbe2Leb6x43mznEzZLPbo29+5dcpVERQazVUxaIQj49IRtDdstHL OoLO+fplycQ9LG4McZqQKAN4i0us9V5zRljNfv5Wn+ryq0Zlm7wyNKzz4+YuOodrRqbm cLBct6RtzX3T00eJgaT7kUcMmggi31CgcgTVAg4AWeJMJwYPmdQl+MtfLdmrCFHSBwQy fyPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VmvynFoDx+JpYeqIScffxYLG+oVJVOqfxROPxE6S540=; b=QY0r6Q3RkB2yZBPeV9E0CvpgMmtcCrSb2KcIc3oKdbDDPi3sR7INyFRZE4DBekfSBN 8aIET/jj/WRLSgT9fRixhuWdvvzG3kGwPq2vbRcNrxLuAeGtkV2B9+iyGPgGn6xfeWDS F6ECepNfYYz71oWkW9O8MvOyY9taYaJ+ofSpDcfiqKWq/sAa9Z1t29dvdpeGxKYRi7ZF xKMWCf8RONZP6dxvcSOO9ScLXd5i63ITjEHI5sttuwnluP3HEo4fZHdCjcMTFEJdEhr8 rpywQ1mL2WyVv8WH+cMRdCjhc70pn+Y73DLBWCIHTe/H70tUrJuKWhpreWJj0NiSfPTG yrrw== X-Gm-Message-State: AOAM5303VCwmbhw8tocBi1sPv/P6y2pK5gX5ihf6+n3RTMYuumyAdqHX Dd3PKJ4rml1ccRgI6TqhMMx1ZMTPxGsZ6w== X-Received: by 2002:aa7:810d:0:b029:363:7359:f355 with SMTP id b13-20020aa7810d0000b02903637359f355mr27258109pfi.64.1628077925583; Wed, 04 Aug 2021 04:52:05 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:05 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:34 +0530 Message-Id: <20210804115138.64475-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 4/8] [GSoC] lavfi/dnn: Async Support for TensorFlow Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: nlEnaKEGgBky This commit enables async execution in the TensorFlow backend and adds function to flush extra frames. The async execution mechanism executes the TFInferRequests on a separate thread which is joined before the next execution of same TFRequestItem/while freeing the model. The following is the comparison of this mechanism with the existing sync mechanism on TensorFlow C API 2.5 CPU variant. Async Mode: 4m32.846s Sync Mode: 5m17.582s The above was performed on super resolution filter using SRCNN model. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 121 ++++++++++++++++++++++++++----- libavfilter/dnn/dnn_backend_tf.h | 3 + libavfilter/dnn/dnn_interface.c | 3 + 3 files changed, 109 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 805a0328b6..d3658c3308 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -37,7 +37,6 @@ #include "dnn_io_proc.h" #include "dnn_backend_common.h" #include "safe_queue.h" -#include "queue.h" #include typedef struct TFOptions{ @@ -58,6 +57,7 @@ typedef struct TFModel{ TF_Status *status; SafeQueue *request_queue; Queue *inference_queue; + Queue *task_queue; } TFModel; /** @@ -74,7 +74,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; - // further properties will be added later for async + DNNAsyncExecModule exec_module; } TFRequestItem; #define OFFSET(x) offsetof(TFContext, x) @@ -88,6 +88,7 @@ static const AVOption dnn_tensorflow_options[] = { AVFILTER_DEFINE_CLASS(dnn_tensorflow); static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_queue); +static void infer_completion_callback(void *args); static void free_buffer(void *data, size_t length) { @@ -885,6 +886,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->exec_module.start_inference = &tf_start_inference; + item->exec_module.callback = &infer_completion_callback; + item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { av_freep(&item->infer_request); @@ -898,6 +902,11 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ goto err; } + tf_model->task_queue = ff_queue_create(); + if (!tf_model->task_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -1060,7 +1069,6 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q { TFModel *tf_model; TFContext *ctx; - TFInferRequest *infer_request; InferenceItem *inference; TaskItem *task; @@ -1073,23 +1081,14 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q tf_model = task->model; ctx = &tf_model->ctx; - if (task->async) { - avpriv_report_missing_feature(ctx, "Async execution not supported"); + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { return DNN_ERROR; - } else { - if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; - } + } - infer_request = request->infer_request; - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + if (task->async) { + return ff_dnn_start_inference_async(ctx, &request->exec_module); + } else { + if (tf_start_inference(request) != DNN_SUCCESS) { return DNN_ERROR; } infer_completion_callback(request); @@ -1126,6 +1125,83 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * return execute_model_tf(request, tf_model->inference_queue); } +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TaskItem *task; + TFRequestItem *request; + + if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { + return DNN_ERROR; + } + + task = av_malloc(sizeof(*task)); + if (!task) { + av_log(ctx, AV_LOG_ERROR, "unable to alloc memory for task item.\n"); + return DNN_ERROR; + } + + if (ff_dnn_fill_task(task, exec_params, tf_model, 1, 1) != DNN_SUCCESS) { + av_freep(&task); + return DNN_ERROR; + } + + if (ff_queue_push_back(tf_model->task_queue, task) < 0) { + av_freep(&task); + av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); + return DNN_ERROR; + } + + if (extract_inference_from_task(task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + return execute_model_tf(request, tf_model->inference_queue); +} + +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) +{ + TFModel *tf_model = model->model; + return ff_dnn_get_async_result_common(tf_model->task_queue, in, out); +} + +DNNReturnType ff_dnn_flush_tf(const DNNModel *model) +{ + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TFRequestItem *request; + DNNReturnType ret; + + if (ff_queue_size(tf_model->inference_queue) == 0) { + // no pending task need to flush + return DNN_SUCCESS; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = fill_model_input_tf(tf_model, request); + if (ret != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_freep(&request->infer_request); + av_freep(&request); + } + return ret; + } + + return ff_dnn_start_inference_async(ctx, &request->exec_module); +} + void ff_dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; @@ -1134,6 +1210,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + ff_dnn_async_module_cleanup(&item->exec_module); tf_free_request(item->infer_request); av_freep(&item->infer_request); av_freep(&item); @@ -1146,6 +1223,14 @@ void ff_dnn_free_model_tf(DNNModel **model) } ff_queue_destroy(tf_model->inference_queue); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } diff --git a/libavfilter/dnn/dnn_backend_tf.h b/libavfilter/dnn/dnn_backend_tf.h index 3dfd6e4280..aec0fc2011 100644 --- a/libavfilter/dnn/dnn_backend_tf.h +++ b/libavfilter/dnn/dnn_backend_tf.h @@ -32,6 +32,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out); +DNNReturnType ff_dnn_flush_tf(const DNNModel *model); void ff_dnn_free_model_tf(DNNModel **model); diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 02e532fc1b..81af934dd5 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -48,6 +48,9 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #if (CONFIG_LIBTENSORFLOW == 1) dnn_module->load_model = &ff_dnn_load_model_tf; dnn_module->execute_model = &ff_dnn_execute_model_tf; + dnn_module->execute_model_async = &ff_dnn_execute_model_async_tf; + dnn_module->get_async_result = &ff_dnn_get_async_result_tf; + dnn_module->flush = &ff_dnn_flush_tf; dnn_module->free_model = &ff_dnn_free_model_tf; #else av_freep(&dnn_module); From patchwork Wed Aug 4 11:51:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29236 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3308099ioh; Wed, 4 Aug 2021 04:59:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzOlzQ6IQtJyPE4gRhHuHaIJbEjfZYhP/azpeyvxESgV0oUeyv3jB7KAnyVWcI3BKaGuNlD X-Received: by 2002:a17:907:9d2:: with SMTP id bx18mr8396215ejc.117.1628078356844; Wed, 04 Aug 2021 04:59:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078356; cv=none; d=google.com; s=arc-20160816; b=lztH29bQZOnXrsPdX1O+Elbx+06CQQpEEhd30RcBG4vOkytX+WFy3CVt0OEtIThxqO ZlB11qSeD9T7DejgG/6g4Xuq44QSx1qrGXB9eBoLYlTAq9+cOUgW7T5EhpZtgrqOf+Wv q+WrUDN7TvJvTNVJPeQnVcRe9cRK+Y1VhEYwUVgMHR/MbK5b83xNucofXRfPtxZRutBV +LZq6kjmzyAfLX0bXr+5VFCbEyZgjCwPeQp4EEA6zBhtzZ0DTzCbejYkozW4RohXybCv kQMVURPzrBU0i6fmMlXOefDWFMH3q2ILVJEAA1w3SQ0HUASHSNuRaYaYizOJSqsAfeNj 0SbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=PquoaGuVaabw19GMBimaFa/zV1bHx/p/nPndb+VBQ9c=; b=f0ECIfQylskEeqvg5RXwTtThZg7oYdxTm/bcZTZqWnrSfbIWcg2iT/bDkREwhoh1ko s2WO0iV7QzGiCU98aXebuHR8Lq0iGKiivOEQs+GJM34RXZX9WIY+na1xnCYeMaO0OXHt 1FWnyTy/bEWKVAG1qIJ4TLBu/Ams8gqDruW6TZLeD1dnP8wOTozmsibV2JAwXk5H0XeI EhF7ZIrPGqUHMrbsCEVqWFLDxufOFHOLTM5y+jUjEgHBLpucDYI/GHrO5JqZq6nCGHnj mj2Uz7DGKSY5KTihviBf/Vaeo7VjlvjGZvGNxxhAzAxGsogXf9n41sVKT+RSl7awis+E 42rA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=N6g3Z14M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q13si298493ejz.268.2021.08.04.04.59.16; Wed, 04 Aug 2021 04:59:16 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=N6g3Z14M; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DFC2368A1AB; Wed, 4 Aug 2021 14:59:10 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 95F736807DC for ; Wed, 4 Aug 2021 14:59:04 +0300 (EEST) Received: by mail-pj1-f41.google.com with SMTP id m10-20020a17090a34cab0290176b52c60ddso3152905pjf.4 for ; Wed, 04 Aug 2021 04:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vauU9bb9s1tBNjhXdaNfdF9AKP8/RU27dqP7XBh53/Q=; b=N6g3Z14MYn7acY2ITm/J89d0Zq2oGEFQmUwo+ZvKL3gryFB7p5HbA8irRu9a2Z1OUs 4ZQVnrsAZMXpP8m2MJoYiEYZ8xRlCRnIYP5HzfNYalp9KYE4Ni5OgFpCzsmEkr2JlzhM Sjh5Pz6x4wZu5oFis+zsbJTBHrWEKjGDe3xm/+c3lZq/Tt9X+fcue6Hn/rI1lINRd9Y3 MjW0+ZyArYzTJwu7FdFUyHY54FcUl3fkHiDyrnRrCa1AQ1TtR1oHYA45bXqABAM8t65C Fe3kvgIkXe6GLDhEdA5xtPiU5rX8FCWbqwVX19kd5ktqT6svzcOdrwgoOLovAUXH9MR1 fDRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vauU9bb9s1tBNjhXdaNfdF9AKP8/RU27dqP7XBh53/Q=; b=EgY83Tzs0LcXWcqbNewatfvjhTr8uNrZS3XZWD5C+ydQdZa7v6/Qo/+MAeISwwzhrV qCHIPZ6LBV6BuIDWLZAHn8XrHdbgSp4ZkwQP5Qqv85Wv6pxK6ysLDL64g38bL3LIHH3w 492UaAKjzQ2bx4h+lqw49FKTTINWH5rG379mnZPRVZZX+NspVUMDWu4akueoZnaDALFg uhcyQGq3FcFkCIAwHR9fhDrjLMHtrt9Nq3Ji7AzIhrbrbIxmcycC7BUQUQJWbXxsZxZu USw5lvMEoVCwVrB9pM8G1jhQ2vizMVmO6hnH5OKJdow28CV+Ink6WhA7bgUlod/jNKG+ H3bw== X-Gm-Message-State: AOAM531ogFqzduw/it1WcQ9PhrPpP7g4LUFcPei5K6y5gCXZpdm8IBg8 zQF6DANo+elDK5DI60bpEU10WGxqq3CARw== X-Received: by 2002:a63:1656:: with SMTP id 22mr335628pgw.163.1628077927387; Wed, 04 Aug 2021 04:52:07 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:07 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:35 +0530 Message-Id: <20210804115138.64475-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 5/8] [GSoC] lavfi/dnn_backend_tf: Error Handling for execute_model_tf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: hCjswgMwU7sC This patch adds error handling for cases where the execute_model_tf fails, clears the used memory in the TFRequestItem and finally pushes it back to the request queue. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 52 ++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index d3658c3308..102e91a667 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -172,6 +172,24 @@ static DNNReturnType tf_start_inference(void *args) return DNN_SUCCESS; } +/** + * Free the TFRequestItem completely. + * + * @param arg Address of the TFInferRequest instance. + */ +static inline void destroy_request_item(TFRequestItem **arg) { + TFRequestItem *request; + if (!arg) { + return; + } + request = *arg; + tf_free_request(request->infer_request); + av_freep(&request->infer_request); + av_freep(&request->inference); + ff_dnn_async_module_cleanup(&request->exec_module); + av_freep(arg); +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; @@ -880,6 +898,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ if (!item) { goto err; } + item->inference = NULL; item->infer_request = tf_create_inference_request(); if (!item->infer_request) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for TensorFlow inference request\n"); @@ -891,8 +910,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ item->exec_module.args = item; if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); goto err; } } @@ -1059,8 +1077,7 @@ err: av_freep(&outputs); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); } } @@ -1072,28 +1089,35 @@ static DNNReturnType execute_model_tf(TFRequestItem *request, Queue *inference_q InferenceItem *inference; TaskItem *task; - inference = ff_queue_peek_front(inference_queue); - if (!inference) { - av_log(NULL, AV_LOG_ERROR, "Failed to get inference item\n"); - return DNN_ERROR; + if (ff_queue_size(inference_queue) == 0) { + destroy_request_item(&request); + return DNN_SUCCESS; } + + inference = ff_queue_peek_front(inference_queue); task = inference->task; tf_model = task->model; ctx = &tf_model->ctx; if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } if (task->async) { return ff_dnn_start_inference_async(ctx, &request->exec_module); } else { if (tf_start_inference(request) != DNN_SUCCESS) { - return DNN_ERROR; + goto err; } infer_completion_callback(request); return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } +err: + tf_free_request(request->infer_request); + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + destroy_request_item(&request); + } + return DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) @@ -1193,8 +1217,7 @@ DNNReturnType ff_dnn_flush_tf(const DNNModel *model) if (ret != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { - av_freep(&request->infer_request); - av_freep(&request); + destroy_request_item(&request); } return ret; } @@ -1210,10 +1233,7 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { TFRequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); - ff_dnn_async_module_cleanup(&item->exec_module); - tf_free_request(item->infer_request); - av_freep(&item->infer_request); - av_freep(&item); + destroy_request_item(&item); } ff_safe_queue_destroy(tf_model->request_queue); From patchwork Wed Aug 4 11:51:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29239 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3308340ioh; Wed, 4 Aug 2021 04:59:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyiTr4vlRNMuXB64HCScYD/8L9LHqsA5rRoTskNW05BJYelDn2M0nqdhzOjzAT9AR1AoZwi X-Received: by 2002:a17:906:2b10:: with SMTP id a16mr25339508ejg.33.1628078385523; Wed, 04 Aug 2021 04:59:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078385; cv=none; d=google.com; s=arc-20160816; b=Tj5hCq7e4g4BWYkgnvxpkGkJF7GvyvQczNJH/zaIpCmxEMziXD9j/b96IR0st1kxHy LvbaTAuawZD2i+sQFmEOMm1WZqVH4qIuW/8fh7MRgqojEi1SeVa6N4YF9CTRg6xycaZg IfzejY3IfAZncecoN42/6Q6NWV92KpnFNMISnLFPe/bwup3mFG2JY7lY0fmQoXeuYzjB RbwSsS8mHQAkvqqJeYSp6N/xb+7LlG4QUaOb/q37ghTTU2MHTtMe0aSyGKxjqiwbgR1r 2MNKd1OF1aP9gDD+idi9VmvfX8yIVnUnnRvnVk12fevetlGymwMh4ELZC4/1PTtOZvAJ 2q4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=KfHwMy0bD6ebqItukOzpbVUkbLZJOnQ72zlViXBuH0g=; b=lw/8qKOZ/HIWKrupGwI3vfzY71YJfWPgo94N/CHcz1sFOyi1mLnYvXkKS1jEQwXqv/ 8Ra0BY/rFgIfbd5tLyiF2QBDRN4MRr+E/wkSSlPV0Og87W/P7zI/VeAxPHJ0T/09VE1E GqtPT8C6qyYveUGUhTDy+WA6RlMeUpMhuOCKCOcAzaYVPSSnQ1sRzJREoJI5/01fD9EK jqqErKGW91pPCliRrl8aPoN8vtyOa2luuv8YVzhhHXL/HUYvjXhl326IAVIsLNcbO0Q7 BPIQ5c3UKA3hEftPn195ck6rVT5rWSmTIzrM0xTQUgRwm0ITxcMAsDS2aWtZ5ko95qyr sG0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=Qlcrbu10; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id i12si1941936edc.10.2021.08.04.04.59.45; Wed, 04 Aug 2021 04:59:45 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=Qlcrbu10; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 31EE0689F02; Wed, 4 Aug 2021 14:59:43 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B43076809A7 for ; Wed, 4 Aug 2021 14:59:41 +0300 (EEST) Received: by mail-qv1-f45.google.com with SMTP id d17so916671qvn.13 for ; Wed, 04 Aug 2021 04:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x6U0tprtfKfu21C6PdlWYMVrnFE50Ul36ctx/RJmXEE=; b=Qlcrbu10vFw/XxFUAEpRYeI+z+Lks0DMlP9l1O8uO3UTlY6po4OMMafd1J7pac8Xpl Rodb6BVoydJc6GsL7NuCYjOgX+oTaH2S6c9RxyQHfML8bQeF/QqzxIWcuMT8NDej2HAy GbSc+Y+6T8v3cdWgaeuecDnQ3SjFXJYE4X1ZXgvq0DJ0SRd/41g6s//6qLiZau36AjKU ODBcJufcZytqSSt7asD7sS0yN6C5p4bZnIGW9wugwp2ORRsnDHgrocu5foI67y8WtA8x pwoVItaxsmXJCVLT02Kw9nsf7sNTLe4bD2AfuARy8tdihZ8tqAneCKxveXeCXDSQ3hMK RAZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x6U0tprtfKfu21C6PdlWYMVrnFE50Ul36ctx/RJmXEE=; b=Qa+bJJcGn7LlOrz+jEaNGCVz3kPIgtvjlSWwpyeih9fSKdYnv4MtK3wZ2/Z4HOyGPe QsLVpVOlZA4aLHr5oQjKvHB+iJt+ZOJYhOeEEP+9dooo6TRgzO2JAnsNI+gg2NxQ7gN8 J6YmaMr37z8YqnUqBd6/tRp56m5QQZ2f/y8yDlfAx5eqTDVNUS2+mpXgyVTAfKwKFN5i NIfAdKEM2FakeyQtUkOLnnAl3XbHeEw6MDBBqXqH5HXQJuuY9hwlD2OII9mCgjCI5QnF o0oWqECIbQi8MDlgetU5X/gCgCS/4haTBpd2tEc9tY9Ir8b70UakH077OckEhE7HgQ0g ARfw== X-Gm-Message-State: AOAM532+FZdwtlZwRSxkA4Mg7Js7Tno0AdL9AKOH0qUlIxesMqkImomd iVV6y6WLpqGCfQhJPt1HZBiJZVWBec/Quw== X-Received: by 2002:a17:902:b713:b029:12b:b249:693f with SMTP id d19-20020a170902b713b029012bb249693fmr6173633pls.17.1628077929261; Wed, 04 Aug 2021 04:52:09 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:08 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:36 +0530 Message-Id: <20210804115138.64475-6-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 6/8] [GSoC] lavfi/dnn_backend_tf: Add TF_Status to TFRequestItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: OrjP9VJL5jd0 Since requests are running in parallel, there is inconsistency in the status of the execution. To resolve it, we avoid using mutex as it would result in single TF_Session running at a time. So add TF_Status to the TFRequestItem Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 102e91a667..c3425e01a8 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -74,6 +74,7 @@ typedef struct TFInferRequest { typedef struct TFRequestItem { TFInferRequest *infer_request; InferenceItem *inference; + TF_Status *status; DNNAsyncExecModule exec_module; } TFRequestItem; @@ -164,9 +165,9 @@ static DNNReturnType tf_start_inference(void *args) infer_request->tf_input, &infer_request->input_tensor, 1, infer_request->tf_outputs, infer_request->output_tensors, task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(tf_model->status)); + request->status); + if (TF_GetCode(request->status) != TF_OK) { + av_log(&tf_model->ctx, AV_LOG_ERROR, "%s", TF_Message(request->status)); return DNN_ERROR; } return DNN_SUCCESS; @@ -186,6 +187,7 @@ static inline void destroy_request_item(TFRequestItem **arg) { tf_free_request(request->infer_request); av_freep(&request->infer_request); av_freep(&request->inference); + TF_DeleteStatus(request->status); ff_dnn_async_module_cleanup(&request->exec_module); av_freep(arg); } @@ -905,6 +907,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } + item->status = TF_NewStatus(); item->exec_module.start_inference = &tf_start_inference; item->exec_module.callback = &infer_completion_callback; item->exec_module.args = item; From patchwork Wed Aug 4 11:51:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29234 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3303907ioh; Wed, 4 Aug 2021 04:52:35 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwtZCjxa7SUP9dNPGqA/EmdVDzBA/fw+rWcEdtz0B2rjiVQf9tdw9Vyvy/AGPImebFclwqQ X-Received: by 2002:a17:907:6218:: with SMTP id ms24mr25690582ejc.367.1628077955654; Wed, 04 Aug 2021 04:52:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628077955; cv=none; d=google.com; s=arc-20160816; b=0QBV9aMvbNm5NwYxjpOnygu4K4/pZZ/q2HzA0alzEqSsHRfrAO2zfFjRWQpWIb0y5U GAHbVg8fqh82iHWtMVHGUVq8qNsJWkQP6pJyszRAOfdams+O4owquLxqlQ6euoeuiUKP oF3PyRUSLmQq1QaE58sLUua9NE8hiyTUJeAizTk1+AFOdHbS0xHgqSlMDLMfsBo1nBPj xSnljiaVK7BneKo4V7cJKFvr3cReYpHyBd8VRHQtTFWF5pSq1QpmYXqL3TfWVhYBjjOB qAvROIuh3JTuJWsZUYVuRsioeRo7k/PtpkJsF2yoNSo6fyvfJBgYTgvy65FWOrK4bRmn 8ksg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=gRyICjugDiayJLj2iEvxw20o6U4kNhcX198/MHGBwuo=; b=rkAwgUyzYs5rgTFycmNoX+ohm3Ua/KKw5gNw/hLb8GwPhmoBdPH3GAJAzbe5QGOJzX 0p9E2NJmyU0FAcs7w7Y/9scEO7A2/DQdsRQO6z11hq5BZy/b3OOLA537rLvb+EPQcXYL yT1jIVmIqkLouduee7ACEUV1uyaKSnPon5nCluXR86DK6FSOfIA7ubD5ZzwOOcBy0S7O fouLtah4UHFx8W+eJTC9/KxVUueJYSqaaebXeIn2J0a0jqTQPnbEXZHcbq1Oe7yz6SKB rZx92sDXtSTrn51PiibVRf389nSJpTJjXYuWAEjozF+iFszvjhv5gc6Ek81rxoxa1rXv 7JJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=ia9SUHWI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gt14si1709359ejb.752.2021.08.04.04.52.35; Wed, 04 Aug 2021 04:52:35 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=ia9SUHWI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2F831689F8F; Wed, 4 Aug 2021 14:52:20 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 18252680C5A for ; Wed, 4 Aug 2021 14:52:13 +0300 (EEST) Received: by mail-pj1-f44.google.com with SMTP id dw2-20020a17090b0942b0290177cb475142so8312186pjb.2 for ; Wed, 04 Aug 2021 04:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D4wOKHSoELtJQgdBzUqXxpwkADqTvb2S78r/abmBbV4=; b=ia9SUHWIHAFoTrKommIN2HFRileqP6GbicQ3569xymt3TjKndNTJiFgSW6Qv4ygixE DJM7zadgf86JhZXPE9Y0wJV/7AofoseZg3oWnK01lzmTIF9rV0Oc2r3Y3oHG331ScXji NfiYm1EFvi/wd1vP+Trqbojtq/sMQHrUNiKowHVolyR9pFr+/XbTnL4UkH96WAEbFD7k V6onO/av9ZJsHgV4QD9L3QQbiZAUUpzsVqCLsfstkjowGtf4m0bwyjkVZcAEVS4tndlE Hjm8KmNVulX7lgm4fUgtoSNllddv8xjfQ+WY+mUyYRKhUvXczG4kaD4Wz0b+EpVvTTQP 5k0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D4wOKHSoELtJQgdBzUqXxpwkADqTvb2S78r/abmBbV4=; b=r4fIPvl4LCUxQReaBnyfwAxXGLJpwOpRanHs08gYMxVsPxCCFLlJdYUzkdk3gYPwmZ 3Z6G7ZTrHrpRmK9F4ckLb7B3NS2oGm6xNOJdnB0D7058rHnis84/rQPeoAfIypFzzJOM BN4NMAGNAYK7VIh7bgULXpWUkQi4Z+qqf2Zfpzrnd7i1Wc7u1lVIe+WBdo7v8C+aZ1lP GaGzsbV0eDVQSMnkBDRGv71Hv/CN/ZtJRX7XAbvfYseLBFMMTFUECTfkHXtcvFsTVwQx 3c6lImpEhSktNZCSSaDTpKbP4oNatGjqPtBXRXdDSJ6pb0P4NccUknmkHapTyj9t6zXL e2ZQ== X-Gm-Message-State: AOAM533D+yH2wuyFRyUQu1m1spN9qm0d01ZnE2QjVf90n1U016nHY6hM BOK0X5aLT+9gu1HkBMACAcnxU+kPOU20+A== X-Received: by 2002:a17:90a:fc6:: with SMTP id 64mr27819050pjz.1.1628077931295; Wed, 04 Aug 2021 04:52:11 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:10 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:37 +0530 Message-Id: <20210804115138.64475-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 7/8] [GSoC] lavfi/dnn: Extract Common Parts from get_output functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: hf+9KNUO+Alq The frame allocation and filling the TaskItem with execution parameters is common in the three backends. This commit shifts this logic to dnn_backend_common. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 26 +++++++++++++ libavfilter/dnn/dnn_backend_common.h | 16 ++++++++ libavfilter/dnn/dnn_backend_openvino.c | 51 +++++++++----------------- libavfilter/dnn/dnn_backend_tf.c | 42 +++++++-------------- 4 files changed, 72 insertions(+), 63 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index df4bab85e0..470fffa2ae 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -142,3 +142,29 @@ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **i return DAST_SUCCESS; } + +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx) +{ + AVFrame *in_frame = NULL; + AVFrame *out_frame = NULL; + + in_frame = av_frame_alloc(); + if (!in_frame) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + return DNN_ERROR; + } + + out_frame = av_frame_alloc(); + if (!out_frame) { + av_frame_free(&in_frame); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); + return DNN_ERROR; + } + + in_frame->width = input_width; + in_frame->height = input_height; + exec_params->in_frame = in_frame; + exec_params->out_frame = out_frame; + + return ff_dnn_fill_task(task, exec_params, backend_model, 0, 0); +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index e1d678b230..604c1d3bd7 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -137,4 +137,20 @@ DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_ */ DNNAsyncStatusType ff_dnn_get_async_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); +/** + * Allocate input and output frames and fill the Task + * with execution parameters. + * + * @param task pointer to the allocated task + * @param exec_params pointer to execution parameters + * @param backend_model void pointer to the backend model + * @param input_height height of input frame + * @param input_width width of input frame + * @param ctx pointer to the backend context + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if allocation fails + */ +DNNReturnType ff_dnn_fill_gettingoutput_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int input_height, int input_width, void *ctx); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 47cbd48ba2..c825e70c82 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -640,10 +640,15 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu OVContext *ctx = &ov_model->ctx; TaskItem task; OVRequestItem *request; - AVFrame *in_frame = NULL; - AVFrame *out_frame = NULL; IEStatusCode status; input_shapes_t input_shapes; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; if (ov_model->model->func_type != DFT_PROCESS_FRAME) { av_log(ctx, AV_LOG_ERROR, "Get output dim only when processing frame.\n"); @@ -669,51 +674,29 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu } } - in_frame = av_frame_alloc(); - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, ov_model, input_height, input_width, ctx) != DNN_SUCCESS) { return DNN_ERROR; } - in_frame->width = input_width; - in_frame->height = input_height; - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; - } - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.nb_output = 1; - task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } request = ff_safe_queue_pop_front(ov_model->request_queue); if (!request) { - av_frame_free(&out_frame); - av_frame_free(&in_frame); av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto err; } ret = execute_model_ov(request, ov_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; - - av_frame_free(&out_frame); - av_frame_free(&in_frame); + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; +err: + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index c3425e01a8..9c96868fab 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -309,36 +309,20 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu DNNReturnType ret; TFModel *tf_model = model; TFContext *ctx = &tf_model->ctx; - AVFrame *in_frame = av_frame_alloc(); - AVFrame *out_frame = NULL; TaskItem task; TFRequestItem *request; - - if (!in_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - ret = DNN_ERROR; - goto err; - } - - out_frame = av_frame_alloc(); - if (!out_frame) { - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - ret = DNN_ERROR; + DNNExecBaseParams exec_params = { + .input_name = input_name, + .output_names = &output_name, + .nb_output = 1, + .in_frame = NULL, + .out_frame = NULL, + }; + + if (ff_dnn_fill_gettingoutput_task(&task, &exec_params, tf_model, input_height, input_width, ctx) != DNN_SUCCESS) { goto err; } - in_frame->width = input_width; - in_frame->height = input_height; - - task.do_ioproc = 0; - task.async = 0; - task.input_name = input_name; - task.in_frame = in_frame; - task.output_names = &output_name; - task.out_frame = out_frame; - task.model = tf_model; - task.nb_output = 1; - if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); ret = DNN_ERROR; @@ -353,12 +337,12 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu } ret = execute_model_tf(request, tf_model->inference_queue); - *output_width = out_frame->width; - *output_height = out_frame->height; + *output_width = task.out_frame->width; + *output_height = task.out_frame->height; err: - av_frame_free(&out_frame); - av_frame_free(&in_frame); + av_frame_free(&task.out_frame); + av_frame_free(&task.in_frame); return ret; } From patchwork Wed Aug 4 11:51:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29238 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3308041ioh; Wed, 4 Aug 2021 04:59:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyjW2aAnlCqOZ9djClfvP9//JG0jCC7zbEbPMGVyUWawv/bbEpuqnuIxT6Pk2JwGOrmd9CR X-Received: by 2002:a17:906:85cc:: with SMTP id i12mr24965063ejy.405.1628078350171; Wed, 04 Aug 2021 04:59:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078350; cv=none; d=google.com; s=arc-20160816; b=dUpwC9UFRnVdphrAyvr8zD4IFhwl5we1+xecKF2ZsrdTRROPPjDZ7UKlLM2hvi06XN v0a374J4ADJekwX+sG2Gh5qC8leB1Y+LvWxMRPNyYUfDDMvK0EaqCjgjMUlC/R4Jt5/B Gqa8tdIRKl/MYpNRrCGuvz9a7sKJqhM7UoMckw/wRimqr7w2UG+CC4obus+3E2unQGzg lkh91osxYYeM6llqIgN9klmxL0PYl56L0ok6z7PvB7dWGePVNo1oeedWgqBeHOf4S3gu U+ZFn3OUeF7s06YHN0xEBBt6heUsHAYhWxSKLaa6aluSBQTYTr3dc93hNiyqOmugnKCJ kKnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=N8W2OlMokVJ5dfn/FDWicEHHABQboJ118xfqHg1SyW0=; b=jso2jKVY6EVRxRONPyv7pWLKAk1QfBQ/h+6vYxq22GnQj+94sysbIyjZ3rcP4lBvB0 3/FJAeBP9mZ7xemgcz3InjCGMsrkdU2DGpJtDBEbuZ+nzhIjf0HDSAKzIwcrO4F7Y0ij wxTDeUKmdvthiPues2ntgNfc7WifvTvVGM1YCRGCmRhy/UHbMCNN5nuSpqDfjg/TvE2S OgRhuKObcBLYGA/3BNVzEAd22jEX6qF7yLdp0ImL9e9R+ciXg6xGaaUWvG8brAIaJwC+ XM+kD3A1y+XkGdS1EGzCBbOotblqmWUIbGvFyOgoUp/9UuyRM46wAyBCK11T1Bu628zC m+3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=mYc2l8aZ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id rv25si1814586ejb.101.2021.08.04.04.59.08; Wed, 04 Aug 2021 04:59:10 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=mYc2l8aZ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DA9476802FF; Wed, 4 Aug 2021 14:59:06 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id DB1696802FF for ; Wed, 4 Aug 2021 14:58:59 +0300 (EEST) Received: by mail-pj1-f42.google.com with SMTP id e2-20020a17090a4a02b029016f3020d867so3163688pjh.3 for ; Wed, 04 Aug 2021 04:58:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BpRvmxEY/uKnNpz71U+knN1cWsUaMH6abM3kZadvY6I=; b=mYc2l8aZHTKVZvSOO0gJjbhexC63+0DJ/WeETLwMG9mBYBIL7hdxFsQXwtIVal/Ya/ vDtnjs1G2b3ZaVGjtA/G9VB/R4qA3TEyoz2+4bD7tP1Wlnl3db3VlxNxUnLRnsaq3M9U HIZgFVIAzzjUKadI2Iz8US3Yfa50PBEpqs/3yLaF1g+jhqUNNm6H8rK1EmHqNq0q6Ysx AGJpCagAHwhpFICINtnjoL8FXONt8e3bbhunTnqHtKN8j8F2vmfCAkXpqrvKffIENCCd gsaoNElURdO5dpFWFVDPCNku55M+1mfjp/cOKV5FtaHgihF+hfHSS6Sb85tWRhtm6CUe YCPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BpRvmxEY/uKnNpz71U+knN1cWsUaMH6abM3kZadvY6I=; b=j9MR/J3Ta5t7ZDgBUs+DW2JtEKgMupz0/eSFaQYgnxbLESqjSW1bE6SsteIqRhXasa Z+zltkqil76/4eawkwO2CigWX4AnrdEbsQUhdbtzW9lMyumAI46t0w3TejChio5rXMHE 4lgqh+H7yefTjcQxNvhFXykJKgqjufKBvmL2514+mucPnwMISEaScXc6Jm2wLVs1bTOa ubedvPTwyEK4Y4275Anp8DmyTmrm7x8Wnhp0yty9GilmG4+PdbUp9W99AAnQokz2dfBU 2X8ubLCoaboreJOwFYRcuELgk8A1oFw9gY3/EyhevYQZgBoV3Q5U4tcbk9L3dBjly2R7 B31g== X-Gm-Message-State: AOAM531gS4vzR6Uc2aZyv6cxJvFmR4NxRTV8J7ZcHIdrwBONjulikpAC JnojIl36UAOa5UImJ8oFppJ/RnZFFouYCg== X-Received: by 2002:a17:902:eac1:b029:12b:8309:1196 with SMTP id p1-20020a170902eac1b029012b83091196mr502077pld.41.1628077933312; Wed, 04 Aug 2021 04:52:13 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.52.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:52:12 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:38 +0530 Message-Id: <20210804115138.64475-8-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210804115138.64475-1-shubhanshu.e01@gmail.com> References: <20210804115138.64475-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 8/8] [GSoC] lavfi/dnn_backend_tf: Error Handling for tf_create_inference_request X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: h8lbTS/TXyBb This commit includes the check for the case when the newly created TFInferRequest is NULL. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 9c96868fab..b14b621bee 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -134,6 +134,9 @@ static void tf_free_request(TFInferRequest *request) static TFInferRequest *tf_create_inference_request(void) { TFInferRequest *infer_request = av_malloc(sizeof(TFInferRequest)); + if (!infer_request) { + return NULL; + } infer_request->tf_outputs = NULL; infer_request->tf_input = NULL; infer_request->input_tensor = NULL;