From patchwork Wed Aug 4 11:51:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29233 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6c0f:0:0:0:0:0 with SMTP id a15csp3307106ioh; Wed, 4 Aug 2021 04:57:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJypVMM+/0+qyCL76WloHUJJsryEuZAk9UHDDxZkE9P3QBbwEfFErrV1zgR3EEa4AG6clu7C X-Received: by 2002:a17:907:d09:: with SMTP id gn9mr25712009ejc.447.1628078253831; Wed, 04 Aug 2021 04:57:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1628078253; cv=none; d=google.com; s=arc-20160816; b=v0mFrGQ8eF4TTnsA/wdM8kKcY+RkWmBrn9dXXd4HqVVpEokEoWqXp2iwPJ2gZnkwQp 3EWgGZHyDFeUvRmw5ayBlI9bsd9ECcoodZTlW4Ro7ebT/e74JT+ObU5o/iqpsAFF2ssM MMT/716bdr4i0CrcVGoNdgZ5xsMFmLA2XTs4TzhNfhMnU3yip47pSogwN9ywXBf7Zw5P yoZhkbNwxCUxOMzg12h3/pHyXnNOJOoPV9YQLkuMhq4DNBa3VnrjS6pX8qbY8x0MCXhK 6ijbScnTK0u14ltGu9zI9XSV1Q3p8b4ndcWnnMeTW6CLfAUvphZ3syIdLXeKE1J9cQ3I 6UKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=eBVHHWZ1lgEttxcEiYZxSDvSzaC5iqBfqr8PEh6kF0DhTYTpp3dPnirhk1Ju2kdzkb d81npa/lOINpkpYlo114368z1qYghVqGbvCiC7sL39cspx2gDWGhTs0nOgbWOOp6XsTN WCeZv4FoYZ43RZdDUT7Hqs7PvKfM5ITM4oL1lBMdwzl7QbL3jnFa9FixFVAbo5GRiaa/ cJk1GQghhDE49LCAJaoOuYPdl4CEsvRtbVEqi4wbv6NNBNd+EiAgzSbzZYBumN2Zx6H4 KiJrsxWHkqjdtY1CiyMeZKFefl0L2dbK5AYNkAnzTYD9jHorNx5tqJOKKGcgogFR6fKF PaQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KyPUNnn6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bj22si1976513ejb.53.2021.08.04.04.57.33; Wed, 04 Aug 2021 04:57:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=KyPUNnn6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8F3EE689B19; Wed, 4 Aug 2021 14:57:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7F18B689B19 for ; Wed, 4 Aug 2021 14:57:23 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id l19so2639440pjz.0 for ; Wed, 04 Aug 2021 04:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=KyPUNnn6frBA3Fb5o8YYa+8M2N2QAFXeWKlMCUkB+tvqKfg2zm3YMNCob0Qxpg+SJ2 h0lWWdOvb2thVd/nTFADZzBLbMfLl+SWI/xbcXIPx4GENVeQ+sXEHvwhWvodZulMfZNy 0KXXbbFD2S8gndqSxm09LT8DLZbj0m40BCpS0KfZruEkMTgM9z3LRhAxn5LovoH9DyVd HlCblIeS59JASqmy2U0Rla+ulle4BlNi1GYhx2aiSCGLvwNFisCojju1sqIVo7tGSIMU 3jltpTm/X8Dmss+HJXY/i0YPwcCQy4cXRkoO4/cgtzgDdO42cmnSxfbka6zQfyv6qCA0 Rang== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=prc5cJT5SU0MmmNUUBHM0XYPrR5o1Ce7YQZgCyggEY9mHF67G5Sf5vCruA/wU+GzJk Dz0+53fFCvAB3+KByl3iscGviv8xRW8XSk19mLv1Jn0QFQknx9p2RvmSPKDJVtUEwH1o fY+DrLp5kX02XO4KowncojGqefwcY0z5s9OpcoADcptRXsiAXr1qxqiE2yYgAfVeNC2y ddkl1lLR0nemhquUvMyEYFDRz+pMRpNTnk3kKPXuHJ3Dl54QnSyU6GT/VWeLbscb6jc8 Od2SU3+aANSQ/7Wq2srB7Ic2PDA2uC+91nyhUUtwut2Q0SfDQL49G43iupu6keFBFKwS 1EhA== X-Gm-Message-State: AOAM530OYspWE32/pW3ve4kC6Rp4vaXyLGNL95j7WhZcunagGlupJ9kD QqPpbV2QVzK5WdGDp1Wa7AboPY6LyErLQQ== X-Received: by 2002:a17:903:248f:b029:128:d5ea:18a7 with SMTP id p15-20020a170903248fb0290128d5ea18a7mr22573843plw.83.1628077919897; Wed, 04 Aug 2021 04:51:59 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.9]) by smtp.googlemail.com with ESMTPSA id ms8sm6040305pjb.36.2021.08.04.04.51.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Aug 2021 04:51:59 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Aug 2021 17:21:31 +0530 Message-Id: <20210804115138.64475-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH V2 1/8] [GSoC] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: ZeJwBXbYFZHd This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif