From patchwork Sun Aug 8 10:55:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29346 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp1980742ioh; Sun, 8 Aug 2021 03:56:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyq6PJf8HRvWGtx4OwJ88dP9eeQSXijQaU5YVy42/OigFH36Nk2vN4z5XqE8y/qMfiRc4gY X-Received: by 2002:a05:6402:3551:: with SMTP id f17mr8099443edd.261.1628420180172; Sun, 08 Aug 2021 03:56:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628420180; cv=none; d=google.com; s=arc-20160816; b=mrKMoyZ2wI0EKEgPWUzNp+7NmZ9yGl/ifNjwH5/cUix1rJh/4x8GyPn7LRwnHYbv1R GJ9JALQAyZhUt3vbDAx3o3IGFsLDWinF5+VA68bsuZml6tOKLCRTgfoGIwdeuRM9xGjE Glq9QKrGwZNmB9S6mqje1v2hxHaFGZA+lyDhHVR224xEy3zMvAphZN7zmQb+Dt4r1Ov6 ltMXVmLjBuKL5C4Xk+e8T0JtnVSUF7lLVxf+64qi0X+whPvgwI8UymQsPgUM384VoeYU KYvajffGaTtUKIn4BAPDDFrHb/X4JoPieMQuG8rlbrvNG2Zy3vQf69HzQ2P9W0li0pzm uJ9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=XXNyPSZOvD+K8CoBc8u+zWJvCAk8Z5rN4jrX0qPogAKZacOJjUg3hgvlNvYCE4UZ2E O76IJslStpLGmXmHVU0pzLPCBwzz5Rp170GH4pPSnFhQ5/OV4csy08bbNXG4CzIiIfnq T/mrl0LVEz52t7JUcuCM4S8ZNx82ay+sVCiOJ3N0cjDyCGKQ3ggYQrqETJODuo99/roU ZnHh6WO93Jh6oeGj/LW4pRHsRnlmN1j5Jspdyw5SByU1ed4LRKjqw4qXqEmLby3mZKah MSHuUQHtRn459ibXZBloQghGVIwPTciQjbbW2hjAjvalaf/51PGL7jlHOVb52iEXsWBM Fs9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YCGI24oS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l15si10622566eje.462.2021.08.08.03.56.19; Sun, 08 Aug 2021 03:56:20 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YCGI24oS; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 68B1D68A404; Sun, 8 Aug 2021 13:56:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A1E0768A000 for ; Sun, 8 Aug 2021 13:56:09 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id cp15-20020a17090afb8fb029017891959dcbso1989034pjb.2 for ; Sun, 08 Aug 2021 03:56:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=YCGI24oSj/b+7Yx33gjdW6LGhHPohlPl227lJeeWlw5yLiYibg/iZG7SSA27FEY4ZS dstJkLoSvLsPvxwvgGcwrwaUhiXfGA9ZOW07MDmYggV/M2Y9puKOY8Xulb6aO8S6Tc9W pcDPfdPdSofpx21ItUViK0ACTsRMNfrt5GCT4WjgIisg2I2ccEsN1VyfQXlHmnDRDMuS q1fyiRN6FKZSSIQB9LzEa7Mom+l9PFlYIsZZ+yD8yTNvOZDR1jlkNdpWuuqX4aJrB+E5 ++IYM9sy2wyDjK8VCu2SubJNVPRH4Jp3e/BDWE6ooQzpzFjnIHZ94Wgr6VkkqQ6K2mmT kQAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=OUBoVrQDoVfSgl0cU01btBp4keAMyPhLh5lmWM6f01glLQ2fqCAP2LFxB6rQUk5YNu G0ER9MthuWZnsof1IQDSBlYmuIcZ14ZV9Eabp2nZB2Sz4nsn7Fr9S4VpA95Qufz7ElrE hgQcciu7rp/dD5rzBBFRQPeLX0nQtmvcc3bjZ3/u/byTujscEAuGBa4brpQU381kZLy+ nkDe5sP0D6BDc2sBw6eF6+bEykQ+kaCFuZ2e3xpEea3wyhxhFMMG3AA7EJuChCmPn3O2 fE2QwobZyQXmh4X2wc8i3gQFqfUBjTDydPA/9iCkC7KQCVKmNi34oZGoQkkacDIGXcx9 l6tw== X-Gm-Message-State: AOAM533GF4aFGo+frzafPIcMQse77pZvmQ6OpyKmmX7hBDw7cFaByXRO mF8IU3xt/jaB6s8TXYE5yJ0rK8sNwB7eaQ== X-Received: by 2002:a17:902:8f8b:b029:12c:d085:7918 with SMTP id z11-20020a1709028f8bb029012cd0857918mr16666866plo.10.1628420167564; Sun, 08 Aug 2021 03:56:07 -0700 (PDT) Received: from localhost.localdomain ([103.133.123.69]) by smtp.googlemail.com with ESMTPSA id r18sm20917405pgk.54.2021.08.08.03.56.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Aug 2021 03:56:07 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sun, 8 Aug 2021 16:25:31 +0530 Message-Id: <20210808105539.18822-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v3 1/9] [GSoC] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: +SJr1ZdRaUNW This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif