From patchwork Sat Jul 31 05:35:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 29151 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp145748ios; Fri, 30 Jul 2021 23:02:55 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzoxo9BNpvvOLS/0g8wqxV3ucqWu3GXLscFeuALXhoor2++btTBQiD4xC1GwPjGhwr1WzPJ X-Received: by 2002:aa7:cb19:: with SMTP id s25mr7595879edt.194.1627711374860; Fri, 30 Jul 2021 23:02:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627711374; cv=none; d=google.com; s=arc-20160816; b=upOrd/jz7ATC4JIBVX25kEkcvR/IkoPEGlE8jP1qgSXattsw3V5UOOcSfh9gU/0pgV yfwSVhEzIPUVZKMBOB3DlQ6Uc5oKYNT1D92XQMCXiWVTekafbxYk9nQILIIn9b26FTMl JwC1Y1McmbbJQRKw1iGwdJ0tVTDa/Q4MgSwDkJukipdmslUlwgSK0Tj1tS2iVIVf45i7 3T7Kn5LcQJnOg4ffuFEMKRwKLNLInTSbHdhRLL/GK+6TfK6K2qAVqUacbyx7rVVj+c3G 7tRiCzHASi949qriD5L/GS6mlTtMtC5EEe81LE6CpVmCjgbpNRFrifOe+VA5dCyAuoe6 qo5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=ccKZNJ5kl5aSPNoDEi7FQ9GJozol7ZkwoZYNUX4XdXc=; b=VnwPfeuGovETQ8gZg72Ivr0N5stiEFkD54JuqY5bBfLl4lzUDFDsnKmXe+gaBtnx4x d4R64aAVEsGe50AL1GeF3oJPT9YxUeWeZyytRXNcWgwZFrTS+Z4wwkuexROONXGSNvXW vCxpeaoc/RC6xMekPIdGD49ZGNMYBtyGPp1Xqv01xSN2sFfIgdwppHEUcw40xyiaN7k6 HQ6MHkuUMIjM2/tRfnsWEKADbtWj5ozFVZqA6I5m1/CNFdVAp1iTl853dd7UU0zs/VI/ TB7khx/JtkYpV8KpFJmI1BeCMUlEjtcFC4N8H0V6Hc8EUC3uEFkJabcKK6407dq4G+Wj 9flg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WrdUt6of; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bl4si1714786ejb.689.2021.07.30.23.02.54; Fri, 30 Jul 2021 23:02:54 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WrdUt6of; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 788B068A466; Sat, 31 Jul 2021 09:02:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 79F24689F1C for ; Sat, 31 Jul 2021 09:02:45 +0300 (EEST) Received: by mail-oi1-f180.google.com with SMTP id n16so9803883oij.2 for ; Fri, 30 Jul 2021 23:02:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=WrdUt6ofGeKK/SPj+dFPy4bjSvtTGI1ffL28EYruF506/FlMVrdqG1OffGAS9e6brJ 0CpV8b2W8Kqa2YhXP1UoHvMcdrb60CQvaysw3tw66WSj/am2LZv3IsDwC+7+N0muzdaW 6DNBRYoWSegWJpEylyS6CCaXH6XuUifrZmVqDQWm8gLzoB/uCQCcnwXDpdEDCVTlQ20A azZSJDESGxXqRWPL3RBP+mykEzrnmNvpX+PCLdU0p1Nw5CdAimMaYRrqQ2HHlvQDYfkH k+0vZNYPrjL401nZGYetkg3j5yNGM0adZMfG6vCuAOp4gXgCr7MNQ1zJMfxekPq2JUB0 8cxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=D5cerQ3/hQylgRn5WLRZ3tyCcBBYxZsxeqLJCkHIZjE=; b=Cis30iefcI468yuM1NSy76vwnV4WKMa/BrB31G9uW+RS4fAppmRQ/zQSva64P8vLw8 y89R3mTVRlkm0BVnzcOn6UloNWB1vdIl3xuhZZnnsaSuE00oYEro7QBcWa5TMPfIb87w n8BeoAylyOqRqwUWSO0ZRyT+AFjnooK4zUY8C5JGFPwzNwwCXKIHnQmdKWTWntaJ+J4i YS2B5GHyCHxnQnX0l8L6E6FS5s3NiWoHH1o6CzRGrke8vsUylJq0l4HtC3Z6Al6nwOoO HEbUl+OvzZ1BPCcj5TX/NEmyD3ZRfbXTPm6JdKxhgc4nk9fPNhXuzFVaPil1Uf1UVtmD BtXQ== X-Gm-Message-State: AOAM533Dl0re8k/ydCPlklBw2JJZ0jKe5ANHdqGHXGDtqGDVDShtYxPW lepnTtMbt1KvtQdfG8LZLBzIdselTANPxA== X-Received: by 2002:a17:90a:7109:: with SMTP id h9mr5640834pjk.35.1627709756225; Fri, 30 Jul 2021 22:35:56 -0700 (PDT) Received: from localhost.localdomain ([103.157.220.250]) by smtp.googlemail.com with ESMTPSA id p34sm4312266pfh.172.2021.07.30.22.35.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 22:35:55 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Sat, 31 Jul 2021 11:05:09 +0530 Message-Id: <20210731053516.7700-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/8] lavfi/dnn: Add Async Execution Mechanism and Documentation X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0I97kkb3YBoH This commit adds an async execution mechanism for common use in the TensorFlow and Native backends. This commit also adds the documentation of typedefs and functions in the async module for common use in DNN backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 53 ++++++++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 56 ++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 4d9d3f79b1..4c9045501f 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -69,3 +69,56 @@ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, v return DNN_SUCCESS; } + +/** + * Thread routine for async execution. + * @param args pointer to DNNAsyncExecModule module + */ +static void *async_thread_routine(void *args) +{ + DNNAsyncExecModule *async_module = args; + void *request = async_module->args; + + async_module->start_inference(request); + async_module->callback(request); + return NULL; +} + +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module) +{ + if (!async_module) { + return DNN_ERROR; + } +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); +#endif + async_module->start_inference = NULL; + async_module->callback = NULL; + async_module->args = NULL; + return DNN_SUCCESS; +} + +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) +{ + int ret; + + if (!async_module) { + av_log(ctx, AV_LOG_ERROR, "async_module is null when starting async inference.\n"); + return DNN_ERROR; + } + +#if HAVE_PTHREAD_CANCEL + pthread_join(async_module->thread_id, NULL); + ret = pthread_create(&async_module->thread_id, NULL, async_thread_routine, async_module); + if (ret != 0) { + av_log(ctx, AV_LOG_ERROR, "Unable to start async inference.\n"); + return DNN_ERROR; + } +#else + if (async_module->start_inference(async_module->args) != DNN_SUCCESS) { + return DNN_ERROR; + } + async_module->callback(async_module->args); +#endif + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 5281fdfed1..96e4df9676 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -25,6 +25,7 @@ #define AVFILTER_DNN_DNN_BACKEND_COMMON_H #include "../dnn_interface.h" +#include "libavutil/thread.h" #define DNN_BACKEND_COMMON_OPTIONS \ { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, @@ -49,6 +50,34 @@ typedef struct InferenceItem { uint32_t bbox_index; } InferenceItem; +/** + * Common Async Execution Mechanism for the DNN Backends. + */ +typedef struct DNNAsyncExecModule { + /** + * Synchronous inference function for the backend + * with corresponding request item as the argument. + */ + DNNReturnType (*start_inference)(void *request); + + /** + * Completion Callback for the backend. + * Expected argument type of callback must match that + * of the inference function. + */ + void (*callback)(void *args); + + /** + * Argument for the execution functions. + * i.e. Request item for the backend. + */ + void *args; +#if HAVE_PTHREAD_CANCEL + pthread_t thread_id; + pthread_attr_t thread_attr; +#endif +} DNNAsyncExecModule; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); /** @@ -66,4 +95,31 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func */ DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); +/** + * Join the Async Execution thread and set module pointers to NULL. + * + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if async_module is NULL + */ +DNNReturnType ff_dnn_async_module_cleanup(DNNAsyncExecModule *async_module); + +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. Completion callback and inference + * function must be set before calling this function. + * + * If POSIX threads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param ctx pointer to the backend context + * @param async_module pointer to DNNAsyncExecModule module + * + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ +DNNReturnType ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); + #endif