From patchwork Fri May 28 09:24:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27971 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp384806iof; Fri, 28 May 2021 02:55:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzg8mjTfMB1OTkuC3pEwfmh71pBs2ace+CqgNjJKMr4bDhYA3zmDyu6jt+leh7lf4a5KepZ X-Received: by 2002:a05:6402:19a:: with SMTP id r26mr8897940edv.44.1622195715211; Fri, 28 May 2021 02:55:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622195715; cv=none; d=google.com; s=arc-20160816; b=V3JSVb8SV/5EOzl2gShj+Fn8EAJ6sXWlgcCp2LUx9sn92z0WpRg5GsUYsn20Kqa1hn Vw09hqHtg7RuTN4asihpC/f+CHcx23qTuph3jLAJKioL0CS+W1Esfk/EitQI9dYz+i5Y xhPLDraLz0A+PPQxM6c8uN2N/sGeJdNJr51VIPR54YkC5+svWgBOk0l8C59cVtPGmWTz MERsz8hqNkh5Vd2KTafh2IZlEV42QIJNRgALzARs8klXI/woLmLKz7jPsmsbR0z/7GYW gCw3b/QVWLb2OI1qjyGB5Ew/saZ7nKGDGWIpVA8zU9nTX8uKThTJWd1PsD9KSdtxjBSd 8vZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=GEo/0/1XrAdrKdht1l/eg+5SqgjoYss5ufrRkgIYRus=; b=cTG6/pK2Igx7R/FokHMstel2lK4tvm6cdmq135OhHIwez7yxbubvIqhiYli88JLRLv eVN5JOR24kKItlZPa2phPYg8q84MhrBus8GwMyWdOaQpx6biDMi4RvqlLM2Ho/T/cwW+ eZyHaGH5ltJJ1Kpl0qguZ+aRCcXu3ujYDxmgHLXUCYN7wnYxlnoUlfyzvg/RhwmJf6T4 SZHiSS4X34pigaqSwI7NBlAaeSn1cNe5eQFd4dDMxRi/sT4yE+xXeWnqqxaifRt381qG u/LnDBWXnSxkrYCwFHSUk3UhDbVlJqE60hfSEanPqdneSQz+K2s4CYkok4KzZ6hyZLAV srCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="TKXTs/wg"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id fi2si4635362ejb.14.2021.05.28.02.55.14; Fri, 28 May 2021 02:55:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="TKXTs/wg"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1D56168A26A; Fri, 28 May 2021 12:55:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 9DE6B680911 for ; Fri, 28 May 2021 12:55:05 +0300 (EEST) Received: by mail-pg1-f169.google.com with SMTP id i5so2182493pgm.0 for ; Fri, 28 May 2021 02:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=TKXTs/wg0W14U+kO3ETpZKVF4FnkIcIUufYyQevK1dQxMQjwH+k3AGdSnp2GCOWNju 99Bhs6P1slEPCX0LpOw1G+f5Q2AZnI5ZwQ1XJuYdpJVYasAn+y8IMYB3yz8l+DNLTh/G Q0cfPTZnZd+N6GcauqYg9z7ZgA9cs1XEXXNzx3uSHYUsrqXvxLRZP+wWEqs5710RJFDS qYqBj6LZTwxVJNRl/9zQb5DnqHpO3zoXwhqOTLwP7hN3vto1dwvgOXwZqPnyzeUBHzul B3maEOnekBpbuaxu8pIb05uz1sOtMX1kyz4OsQ+I/hdUBVTUR6Mfg8yoi3UOkZHEYno3 BPgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=erF11YD3py2f2l5ecb/WCbFCl37zNobmVZU30NJq9VU=; b=WVvMPEwGlWhYKc75RCUcE2876Rr8ZSO3qzVTpcXE9R95R3pK4/UDkmFh0e3TOZzweC ghu3G5j15WHv8Vh+K7BXgFPFq9jBuidBCmdWJCHhwg2ibw8hSm2ViACuSqupaDAtpCpI 33XzsFlLVn/cQZOixIVEwCoQ+e70KcTcEaF5IGvCBcAAbza7Mvv7w0XQAB/s0u9JfJ0F WVvC3WzE4eYx5lNiQKwp503kyMKckTPgZpSdQeALOf/h2V4FH/NSW6fwve61CtG7pi04 YHpgoNfqu+KziL6acvp34MsegUofwUECM2wRJXccnbvqW9pkAUqD9PuwO8TByOHDXV0+ 5TNA== X-Gm-Message-State: AOAM530rSnziRcgaQ0uTbaCJ6mwjXsgYz3Pb5dv5Qlr1q/8zqmXy17TT RvWVrzX+/jFz8aCKkRsazsK+7Wgr91+ZyQ== X-Received: by 2002:a17:902:7887:b029:ef:9429:e956 with SMTP id q7-20020a1709027887b02900ef9429e956mr7308325pll.78.1622193947946; Fri, 28 May 2021 02:25:47 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:47 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:45 +0530 Message-Id: <20210528092454.31874-1-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 01/10] lavfi/dnn: Extract TaskItem and InferenceItem from OpenVino Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 7OTlDQLsG0rq Extract TaskItem and InferenceItem from OpenVino backend and convert ov_model to void in TaskItem. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 19 +++++++++ libavfilter/dnn/dnn_backend_openvino.c | 58 ++++++++++---------------- 2 files changed, 40 insertions(+), 37 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index cd9c0f5339..0c043e51f0 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -26,6 +26,25 @@ #include "../dnn_interface.h" +// one task for one function call from dnn interface +typedef struct TaskItem { + void *model; // model for the backend + AVFrame *in_frame; + AVFrame *out_frame; + const char *input_name; + const char *output_name; + int async; + int do_ioproc; + uint32_t inference_todo; + uint32_t inference_done; +} TaskItem; + +// one task might have multiple inferences +typedef struct InferenceItem { + TaskItem *task; + uint32_t bbox_index; +} InferenceItem; + int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 58c4ec9c9b..a84370d689 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -59,25 +59,6 @@ typedef struct OVModel{ Queue *inference_queue; // holds InferenceItem } OVModel; -// one task for one function call from dnn interface -typedef struct TaskItem { - OVModel *ov_model; - const char *input_name; - AVFrame *in_frame; - const char *output_name; - AVFrame *out_frame; - int do_ioproc; - int async; - uint32_t inference_todo; - uint32_t inference_done; -} TaskItem; - -// one task might have multiple inferences -typedef struct InferenceItem { - TaskItem *task; - uint32_t bbox_index; -} InferenceItem; - // one request for one call to openvino typedef struct RequestItem { ie_infer_request_t *infer_request; @@ -184,7 +165,7 @@ static DNNReturnType fill_model_input_ov(OVModel *ov_model, RequestItem *request request->inferences[i] = inference; request->inference_count = i + 1; task = inference->task; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { if (ov_model->model->frame_pre_proc != NULL) { @@ -220,11 +201,12 @@ static void infer_completion_callback(void *args) RequestItem *request = args; InferenceItem *inference = request->inferences[0]; TaskItem *task = inference->task; - SafeQueue *requestq = task->ov_model->request_queue; + OVModel *ov_model = task->model; + SafeQueue *requestq = ov_model->request_queue; ie_blob_t *output_blob = NULL; ie_blob_buffer_t blob_buffer; DNNData output; - OVContext *ctx = &task->ov_model->ctx; + OVContext *ctx = &ov_model->ctx; status = ie_infer_request_get_blob(request->infer_request, task->output_name, &output_blob); if (status != OK) { @@ -233,9 +215,9 @@ static void infer_completion_callback(void *args) char *all_output_names = NULL; size_t model_output_count = 0; av_log(ctx, AV_LOG_ERROR, "Failed to get model output data\n"); - status = ie_network_get_outputs_number(task->ov_model->network, &model_output_count); + status = ie_network_get_outputs_number(ov_model->network, &model_output_count); for (size_t i = 0; i < model_output_count; i++) { - status = ie_network_get_output_name(task->ov_model->network, i, &model_output_name); + status = ie_network_get_output_name(ov_model->network, i, &model_output_name); APPEND_STRING(all_output_names, model_output_name) } av_log(ctx, AV_LOG_ERROR, @@ -271,11 +253,11 @@ static void infer_completion_callback(void *args) task = request->inferences[i]->task; task->inference_done++; - switch (task->ov_model->model->func_type) { + switch (ov_model->model->func_type) { case DFT_PROCESS_FRAME: if (task->do_ioproc) { - if (task->ov_model->model->frame_post_proc != NULL) { - task->ov_model->model->frame_post_proc(task->out_frame, &output, task->ov_model->model->filter_ctx); + if (ov_model->model->frame_post_proc != NULL) { + ov_model->model->frame_post_proc(task->out_frame, &output, ov_model->model->filter_ctx); } else { ff_proc_from_dnn_to_frame(task->out_frame, &output, ctx); } @@ -285,18 +267,18 @@ static void infer_completion_callback(void *args) } break; case DFT_ANALYTICS_DETECT: - if (!task->ov_model->model->detect_post_proc) { + if (!ov_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); return; } - task->ov_model->model->detect_post_proc(task->out_frame, &output, 1, task->ov_model->model->filter_ctx); + ov_model->model->detect_post_proc(task->out_frame, &output, 1, ov_model->model->filter_ctx); break; case DFT_ANALYTICS_CLASSIFY: - if (!task->ov_model->model->classify_post_proc) { + if (!ov_model->model->classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); return; } - task->ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, task->ov_model->model->filter_ctx); + ov_model->model->classify_post_proc(task->out_frame, &output, request->inferences[i]->bbox_index, ov_model->model->filter_ctx); break; default: av_assert0(!"should not reach here"); @@ -445,6 +427,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) InferenceItem *inference; TaskItem *task; OVContext *ctx; + OVModel *ov_model; if (ff_queue_size(inferenceq) == 0) { return DNN_SUCCESS; @@ -452,10 +435,11 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) inference = ff_queue_peek_front(inferenceq); task = inference->task; - ctx = &task->ov_model->ctx; + ov_model = task->model; + ctx = &ov_model->ctx; if (task->async) { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -471,7 +455,7 @@ static DNNReturnType execute_model_ov(RequestItem *request, Queue *inferenceq) } return DNN_SUCCESS; } else { - ret = fill_model_input_ov(task->ov_model, request); + ret = fill_model_input_ov(ov_model, request); if (ret != DNN_SUCCESS) { return ret; } @@ -694,7 +678,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu task.in_frame = in_frame; task.output_name = output_name; task.out_frame = out_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { av_frame_free(&out_frame); @@ -814,7 +798,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * task.in_frame = exec_params->in_frame; task.output_name = exec_params->output_names[0]; task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task.ov_model = ov_model; + task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, exec_params) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); @@ -861,7 +845,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa task->in_frame = exec_params->in_frame; task->output_name = exec_params->output_names[0]; task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task->ov_model = ov_model; + task->model = ov_model; if (ff_queue_push_back(ov_model->task_queue, task) < 0) { av_freep(&task); av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); From patchwork Fri May 28 09:24:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27969 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp372649iof; Fri, 28 May 2021 02:33:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx04V0RcbCxGuUOywrG8KC6G93PI2nudbzubX7jHczxeqkGU03YqqldsJZaU1qaIvB3K5EA X-Received: by 2002:a50:ff15:: with SMTP id a21mr8871934edu.103.1622194428102; Fri, 28 May 2021 02:33:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622194428; cv=none; d=google.com; s=arc-20160816; b=PmqJOqFEEPEfs/7Y/4Z32mkJ39vHDa5PoXdn5eOGpF20MxdbYXv1XJ1ZUoz62HqcG5 pt+bqB7eZ1Zc8wLARxQ+wOa0JlW4Up1rOhd2Oqnc5YPAbyPbr8iuii3vkE4QkZkSqa36 v9rUvZK43vx0bnwoq5bnZIeHu/Hn0t4wJfNzrM3C3uSRk7C6BIH47EkGiu5rFUnmS8IC zUrO5qG6PsmLBmfdPxCuWscXuZ9hP26TjZKjmSqtvV1hZbtWGS2wIB1Z7IqO6uTWLaFR //S7VvXbhVa81wxW+UvJRGQxcp4YgI0te4aOF6j+jTwNSH/Q01SLilCQZjsMx4x0v8r4 RBAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=CCaIVmiHRqb3kTamUpgAh59WtS2XXtnqUQZx5GjaVbY=; b=JMgnKGKvuX3YYfJ3tcT6mSEfXKqYdStMqLOH1osL8lEJhGKxnAYz3dHb0el9Vag1pS 3r0SFUUSYFRpSfxoxHO/J3QT+8muKbhJIOY8B9kvKfNQlPhMkiHwzvDhRIATBqKlmcVH P4RUizT9Nt30oKAZHzk1QXAFb2pIgNG4rQ91wFQ3p/PIHAOIFLTnXK/ZeRENAzIlPIr1 DURFpJWvIb2GmHYJ0VSRJJpBPm4+PHcjMSLkYvV+KivpUmF1/oPGSMpqNtVVg5GxB7Kp IKFc++07tUjBvcwhsYtweSGwwziDxVq6iyF0kwKObtP9/4YkEH35VYijdC2b0K6Wuebk 5HBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=TzRc+xFh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z13si4199392edb.480.2021.05.28.02.33.47; Fri, 28 May 2021 02:33:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=TzRc+xFh; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D80C668A1A7; Fri, 28 May 2021 12:33:44 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 30EC4680BE5 for ; Fri, 28 May 2021 12:33:38 +0300 (EEST) Received: by mail-qv1-f43.google.com with SMTP id h7so1527789qvs.12 for ; Fri, 28 May 2021 02:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4gxJawNDd2iWOL99+S3dGdz3E8dlMirBOuo6hxiai7Q=; b=TzRc+xFhllt/VmcQwjQSRvfiLKGUSzdpiolC/Y6NCG+dK612gYAZ3iU+dT2TlB9bZE sFOxQzXbKThKBRrb6uaTgnvRzFAlb7hzYQ1HIfbLX8xGIjeJDn27vi5pQd8G9CTaL13l 2FN6q487TfS4FZjWGPzFLk7jrQilPJRSCdJZEhPw2wE6uOscHZ3WwfMohUTMs/NECuSY ob58mX8rh3GsTOSSdL5EKEFmQvecw/QPFBek3BWLUmJt1j5WmTHn3V8Dov5yHaHCXxss 2KkUEcnvcLuhY8UJO5DN8GyjObDHSyeIIdsSSsaELRY9zCyLjjDItd0vNqh7TBANjjD/ KCGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4gxJawNDd2iWOL99+S3dGdz3E8dlMirBOuo6hxiai7Q=; b=lrm6yxk6XrER24sNYxPIKxqTpdqDDBg00jjvX0dXytmkeBAOVxaLV78vtfi+OOlMQr 6arUopoFcSALE3nBuIHhaFHrAYJVJm0miyVT/c0izJVvYT2XzMcO9PyBik3vtHejuUXM +dFI2DgLEa6LmoHd/Ub1ho7WhLJ+dkH4DuUVF0mpuUAJwyp1F38o77l12jUDd5LreN6+ xBwhMfICCKyxrrQYFLwH088EBMUlj1gU3mySzquM3USucYJxupvNKpS/leF9TOjbm/UB spJcjHgsa5bTq6XEcwYjuj/wj95yNmsciYLWiz5IBUsxtdI54anrvqn2Hm3wsBByIRrz evog== X-Gm-Message-State: AOAM532roI4kVOoMaCYr/dh+D74T8YtpkUDw+TXOjG8YKYQL4LkTiLBo YJxsrVrqp1C4s9o2vSKWAmwmBLfyS+ev3g== X-Received: by 2002:a17:902:d64e:b029:ef:8b85:d299 with SMTP id y14-20020a170902d64eb02900ef8b85d299mr7163816plh.27.1622193949762; Fri, 28 May 2021 02:25:49 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:49 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:46 +0530 Message-Id: <20210528092454.31874-2-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 02/10] lavfi/dnn: Convert output_name to char** in TaskItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: DhwVdVckf2w9 Convert output_name to char **output_names in TaskItem and use it as a pointer to array of output names in the DNN backend. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 2 +- libavfilter/dnn/dnn_backend_openvino.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 0c043e51f0..f76a05026d 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -32,7 +32,7 @@ typedef struct TaskItem { AVFrame *in_frame; AVFrame *out_frame; const char *input_name; - const char *output_name; + const char **output_names; int async; int do_ioproc; uint32_t inference_todo; diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index a84370d689..0f3b235820 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -208,7 +208,7 @@ static void infer_completion_callback(void *args) DNNData output; OVContext *ctx = &ov_model->ctx; - status = ie_infer_request_get_blob(request->infer_request, task->output_name, &output_blob); + status = ie_infer_request_get_blob(request->infer_request, task->output_names[0], &output_blob); if (status != OK) { //incorrect output name char *model_output_name = NULL; @@ -222,7 +222,7 @@ static void infer_completion_callback(void *args) } av_log(ctx, AV_LOG_ERROR, "output \"%s\" may not correct, all output(s) are: \"%s\"\n", - task->output_name, all_output_names); + task->output_names[0], all_output_names); return; } @@ -676,7 +676,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu task.async = 0; task.input_name = input_name; task.in_frame = in_frame; - task.output_name = output_name; + task.output_names = &output_name; task.out_frame = out_frame; task.model = ov_model; @@ -796,7 +796,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * task.async = 0; task.input_name = exec_params->input_name; task.in_frame = exec_params->in_frame; - task.output_name = exec_params->output_names[0]; + task.output_names = &exec_params->output_names[0]; task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; task.model = ov_model; @@ -843,7 +843,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa task->async = 1; task->input_name = exec_params->input_name; task->in_frame = exec_params->in_frame; - task->output_name = exec_params->output_names[0]; + task->output_names = &exec_params->output_names[0]; task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; task->model = ov_model; if (ff_queue_push_back(ov_model->task_queue, task) < 0) { From patchwork Fri May 28 09:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27970 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp380860iof; Fri, 28 May 2021 02:47:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzrwvwNSScjU6ikyxPJsTNDtHjxWAMp7imh9qbSnyFY4BsiaPKOXM4w3egw2mi7TxRdn4X0 X-Received: by 2002:a05:6402:16db:: with SMTP id r27mr8593348edx.375.1622195275943; Fri, 28 May 2021 02:47:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622195275; cv=none; d=google.com; s=arc-20160816; b=W5xyXth8F3YmGX8J7HR/hLGpnyScVVirqoxY1/AvD/BfDw+ZpZLBbXXizMeqcYNSRo N3xzNcELOe0g0czd/Q1gaisuxe308tIrNhSwqbSbpCA2b95n6PMKGaHXLBmmTCyvajw7 z3YYi46Km3oiKt9fQMaFWX39F51JJd9mKfBwwtvcYjMhx5Ax7mfnf6RYmI5q0SJAFmke 84Ui6CDhH0PVjUI3aJFC8qktJjXXTx0ivDBzppbvCsizg+jIP9OqJNca7hDfT7GtM0lt FRNAa60ShZMB0Onb4M2BRMUC+kswXknreM5IIfIouBgTH4vtvb3xzlLK7+vjz5TcneMU h2YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=emymKabnFv5Ls4izyJf7J1g0ntecFJkic0zFGt1/Iy0=; b=IaxPZ3tvueq2v2c8DvnVgn+Wuu461fTeK73n8qCJ5dzywtvOEB8fgc8/N8E66M3RLH k0PDZrL9NmFlTyC0VE9VN1KmmHzytrAN3s6iBqm8fY2awif54FbhLmvmE7hMT692NUru eDzkOt8+AIG9LO5hM6FdDLRgORAhQXagYdB6vCECdqPbsLPiSm24hBYXl8ux1c1wxDKd 2rAaDnuYfLmZDqQ4c8AmBCY6mcx2yo39B69H7Ie+4IdhONPPvICDz4yABLjxCprAVw3V DstBGNEVJgXO0+e9oDMd39TlnqFrd9KQVd/kkjmaecvptY2qSjdCBNJ0mKUb76aUjwoU USZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=SZ0YF3BR; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m7si5040296edq.235.2021.05.28.02.47.55; Fri, 28 May 2021 02:47:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=SZ0YF3BR; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 502A468A214; Fri, 28 May 2021 12:47:52 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 5C42F68981B for ; Fri, 28 May 2021 12:47:46 +0300 (EEST) Received: by mail-pg1-f176.google.com with SMTP id f22so2115000pgb.9 for ; Fri, 28 May 2021 02:47:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mFZCaPaO/WEp3G0w6D9geo+ojGtEwjtWaW8n6GNXQj0=; b=SZ0YF3BR1YCdequQNDuW0DVhPqytU6iZhbSCXmqDxKrm5qgiRJkQtsCSeQJ/fRAsl0 DrdG8aOLmrFrMW1kiJ8s540MWzyamlpDujWr+P3OkFbEMl3Hh8M1KBrBlXzrLI5+C1ta OqEeIM9KG9nRo15jBOn9nHBJjdyE/V8KzZ+flcd/yLZRAlEad6TBg+C4WaXjbL9vyKVy yzhkXWaACAJNJm23fmAod5ena20TpRxsi+5mHUOxwMpIOJe6kdXqT8r2/bBD6W6UKIav 3r10Mz31rcwlQvRTNaOFae/yr0/cIgDnBsf15vV26UYfKzu8FMCQpDob+pebq5cNGy4q HJ5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mFZCaPaO/WEp3G0w6D9geo+ojGtEwjtWaW8n6GNXQj0=; b=LHWMlYDffXq1MoT+iSSdJUwkRWa/wWe5XRQBjwbi2uCzLCl2EAdN8QhuEWYJs7jKol 7s9qmv63b0jpnd0kc7UWkZxUg246Asq47Crr1rFtzks+EmHjVjjXKMSmQCjXZskvdq+o wofdA4vA7s2yMGL0mp/IrIaQPDmGh+3X5FoA06iCwYHF/XJ/xfWZtcLUvItEhJfmQw84 33KtC3k4kQVfuujrZhO2hXBI/jf66UYLVBxLeTKsFn3fgGnOQnJgilcmQfbOH85txpdY 27HNr1sfxJBRF2k217A9CoUfw3mCweFoPe1XT3q0dzyLkA5Ksyvt2waNuvyU90ADj3oi phqQ== X-Gm-Message-State: AOAM533N8I+CX2xiNEkVsD2hUkE3d+rAe7oVoGBpI1QqH1mPQY6wL8cZ wkDshHrbvt5RWWP1kYlV+Q/CbZ3Ok+Arvg== X-Received: by 2002:a17:903:2c2:b029:101:9c88:d928 with SMTP id s2-20020a17090302c2b02901019c88d928mr106086plk.62.1622193951579; Fri, 28 May 2021 02:25:51 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:51 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:47 +0530 Message-Id: <20210528092454.31874-3-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 03/10] lavfi/dnn: Add nb_output to TaskItem X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: YLpxXLnEfu6o Add nb_output property to TaskItem for use in TensorFlow backend and Native backend. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 1 + libavfilter/dnn/dnn_backend_openvino.c | 3 +++ 2 files changed, 4 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index f76a05026d..704cf921f1 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -35,6 +35,7 @@ typedef struct TaskItem { const char **output_names; int async; int do_ioproc; + uint32_t nb_output; uint32_t inference_todo; uint32_t inference_done; } TaskItem; diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 0f3b235820..c2487c35be 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -678,6 +678,7 @@ static DNNReturnType get_output_ov(void *model, const char *input_name, int inpu task.in_frame = in_frame; task.output_names = &output_name; task.out_frame = out_frame; + task.nb_output = 1; task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, NULL) != DNN_SUCCESS) { @@ -798,6 +799,7 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * task.in_frame = exec_params->in_frame; task.output_names = &exec_params->output_names[0]; task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; + task.nb_output = exec_params->nb_output; task.model = ov_model; if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, exec_params) != DNN_SUCCESS) { @@ -845,6 +847,7 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa task->in_frame = exec_params->in_frame; task->output_names = &exec_params->output_names[0]; task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; + task->nb_output = exec_params->nb_output; task->model = ov_model; if (ff_queue_push_back(ov_model->task_queue, task) < 0) { av_freep(&task); From patchwork Fri May 28 09:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27965 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368318iof; Fri, 28 May 2021 02:26:04 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzeIl3zx6Ie+/lkownEMxhpWCYnJ3ebQaZoMbiWCRetqES/8VJ/j15Vym98FGUh1w4LuX4a X-Received: by 2002:a05:6402:42cc:: with SMTP id i12mr9168427edc.272.1622193964283; Fri, 28 May 2021 02:26:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622193964; cv=none; d=google.com; s=arc-20160816; b=Zqq30uiE69hClH7onSNemJjabQ7ZwDMR1ff3sBTRLbhbj7ite+ag09tOAs0a6nbZbX 1uT/rPAdQiJQqDENVO3av6C1XFSzdaTds8zVifEg/Dap8vGPnFNkD1fntLrAgTnv8lf8 r7Cz6iU8oHGNgSuupQOQwY6f/OQS4CGzOejZD6vm4PB4xzLzAF4ssy84nlcWRR/m/ihT 5WyHOpG8cLI7R/txKqA0MQu5fFp3kqbBkxfvtWfgBEbW5Q9/BopyiijlVS77/NchOx0r 9XRqZYhJgjh0tZ7A2jjUswZGL4PDqpnnbDhcxFceOGLR0XA2NFSPDNluB0cG3VKgUO0b Gt4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=TxcIZCuAqi/BslpgzEwZRB9LDj4xcaJSc2mP4bD8kvI=; b=Vp8anDnlttLSfyoGmSh2qlXtmlJmCnaD7oEQ9OCgw4R7PjBa0OcBU+wPdqPgQ07Xz/ 6LpGAP8+VcOET+ifZGDGPfQVtPiPiSDBsn9TUImwg4VedetWlkxedkhgKlAHeB8Guq2X pmzxZ1RQA1D9rHPPSZBAmJglQ1zCTbbawZnBo8hw/yDm0T28357W6dUNo+9zS2++/XlW O6KK6fi7U5hNfi3sLS/sfXWywi9MsMe2gtXf5QmqMcSnHwoj5ZDPdRYByd8Fx9eZMNQd cZPdaarG9XbLOzaxqzkgJauoZUSvRB62bwFiF43UiItBxCOxt5PuOlrY80/1XYlscE43 m0nQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=UetE9JB3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u24si4456554ejy.37.2021.05.28.02.26.03; Fri, 28 May 2021 02:26:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=UetE9JB3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0A2B5689E7E; Fri, 28 May 2021 12:26:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1961A6809B7 for ; Fri, 28 May 2021 12:25:55 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id lx17-20020a17090b4b11b029015f3b32b8dbso4190158pjb.0 for ; Fri, 28 May 2021 02:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MVayHqylz7jNiZJI9/HCLY9SxiXlUMBUuBz4k4NNPG8=; b=UetE9JB3LOLkoQB2iPgTGh2ZdZnuXc2umshEDZV3bk5b3F8QaQqvZ1CzuzqZwNO13z bIJfFeuG8v6OcxpZUtmPYsxy/S03FZvFuJqka+wJwHuoDVTzqG0Jllq1HZrvSWWQKDf6 bqUbbWgzu+0Ly1KdZrhfTk0k+RK3nmzygEr2PQGo1EZXuAl/CzY64Sf8PhaG91lcCfk/ WiEJ90/nQVuPKRPSlTkzF9N9DIBXk990/HumbKDsLB1sD/Gt3Z4cZTOkQ70SjB0296Yt 5YtOUdRtF6iEGXmATZsopbSjtCudjP1ROk01wiJtQMSBE64iSAGIxPdYSxEvS7564+gl Afug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MVayHqylz7jNiZJI9/HCLY9SxiXlUMBUuBz4k4NNPG8=; b=o8GLqTnECmvnznSiBssJqcMi57SJsZN8IaDppmXgNQjaZnHZ1+HoGDW6uKEz2vzgEP eiHCwHKKFc0mxhS7eFm4mw/gAyZLETbr4dO5QSnkGaYuNfRf8PO98d6FRYV1GdLOcyk3 COkq945huV5S5DTA81tn7nBeNUrCwqzqnFyzuPJSdb/10ha1EWM9131rS/bVyW7s7FkX cLz9sw4ITOzO1rL6ErDpXqeirP9lyigdVUeatGLGSR5MksV8YBwIeaIR208UbfaRTytY vqpUtiFqVPA+EgUB70sysTcdDYqpV1jd50yzH7BDRXe/oQdfP8AhM5YHFbLnXc1wGHrQ 8grg== X-Gm-Message-State: AOAM532Lp55FKT7CM8r/fiJDZEQiIEC3EierT2RrCXovDKRU2F8oeEm7 DonJRmUqW7OOF6itMi/o3lln7vhkY4G5PA== X-Received: by 2002:a17:90a:7896:: with SMTP id x22mr3505647pjk.11.1622193953365; Fri, 28 May 2021 02:25:53 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:53 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:48 +0530 Message-Id: <20210528092454.31874-4-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 04/10] lavfi/dnn: Use uint8_t for async and do_ioproc in TaskItems X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: XRJhoWGuNzF4 These properties have values either 0 or 1, so using uint8_t is a better option as compared to int. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index 704cf921f1..d962312c16 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -33,8 +33,8 @@ typedef struct TaskItem { AVFrame *out_frame; const char *input_name; const char **output_names; - int async; - int do_ioproc; + uint8_t async; + uint8_t do_ioproc; uint32_t nb_output; uint32_t inference_todo; uint32_t inference_done; From patchwork Fri May 28 09:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27966 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368416iof; Fri, 28 May 2021 02:26:14 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx2NhaRprXuocFQ1Ryd/dIZU9ORqRTm0de6ZvRqI7Ako4B1/bLYxXMCN+pHUFcbcodqeAD8 X-Received: by 2002:aa7:dbc9:: with SMTP id v9mr8887980edt.183.1622193974587; Fri, 28 May 2021 02:26:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622193974; cv=none; d=google.com; s=arc-20160816; b=z/Yx8PnyvVhWOC+cuTvt3k2qhq4qcJKg4ZKCLdf7mzybOUyStOM6pBMyqPy3xzJImU 92H9+njzMf79pdpGEUUTq8PyhgDr3tPgk093YlTsVLbBb276eowSzqJcFWb2ujua1V+L wEiVNpDc/MP8HWBdE7dMHkimMJ91b/wRVOjhxS3hneryIsybY1uprZW2TssTGscZkOM1 p6zvIsClBLn1KJPV8ZRjy4j9dsRgabQdiQkDJRyuza1YxL/bVByPFLXWYoiQKTDdR8iX yWqg/s24lamdF8I1Epl2uIOVkMIaCTOppy5RxbOPrjn40MAYflCw20Hua9k2hivxc/Wu d3ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=enkXEmoLATIGnVyFqWx1KfSuHunLl0HGnAhTwRKtEhI=; b=FQKPRXVIxrLbQS6dgs/rNvgSvHyZLfvocrCCU9ujpfd/2klu1Bd6z5qiKNfGoNgC8C Yo5+3u/SSdlKhCjAD5F0cPtLfnatTkNlAsat2pr0zdp8mPaXQEHHICRBwN1p7U9BTaki ilqdOCCaQnNpmxvEfXvJVgtUaKUlFD4hcoTTKJCSPJFH/RiZ5W7pWYu+pkEGmBvJgrkR FTgEsqdj6AIQO1CWKJlzWo+cHV6zR3vkacVbpJZIl63wX+50jSeIhY/+bO4YsTeEh2+g wF/6z8jVIzPaLoJkqNzz4sbHvaLzDYiFtp2tCt+kK0vB/z0RAeq7pd/3WVwbOQCf7Ant /YNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=EalJYESd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m9si4188798eds.410.2021.05.28.02.26.14; Fri, 28 May 2021 02:26:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=EalJYESd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1B984689EEC; Fri, 28 May 2021 12:26:03 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 0767D6809B7 for ; Fri, 28 May 2021 12:25:57 +0300 (EEST) Received: by mail-pl1-f180.google.com with SMTP id x10so439591plg.3 for ; Fri, 28 May 2021 02:25:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NiWqJtj+P+9dijQdiV2TF8bwccUw6X6PrwK9rCOzSHQ=; b=EalJYESdGShgxrqeYBUNvRN0N5W52bIQSj4+eC75kaVR9q9AMLo5a9Ta/YCtNjLAqF fy7Q2ljjadKwyNZtMOzT/svUMUR9nQGDhHTa5gFngbZxTn8NNFgCiH2WgIliBt6tTm2B KpBxMrfLb7TZDzLjfEhsmIfU1KqC6UNZqcgC3roHIKSAhuaGGr6XIudQ+U/PFB5zN58s waI86/tsbMSXL8sWtIqeFDoQiA9iyqb/L9yXvCVPjcGfcO3jigc73i15iNCRx1fflV+r GfK1+XiWxjCvmbLJtnl6UfqPIHM/ZvaI+RmymP6jQ0N1927R/mz7pV1Sk0ytO1qQ0/d7 E9MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NiWqJtj+P+9dijQdiV2TF8bwccUw6X6PrwK9rCOzSHQ=; b=PkKi19MgQ+vRdZNRfutgUx1xFqw7WeniDKx/a2rCoNfGy96GV6VyndCsL5alECFGdw We1w21KdLEIhH6xwglc7SM6YY+7v54aRCwOIxujfSHi19AgFxVriXXnhV3S71CCn85Wx e+xTi00A1n78JqsR3I63wWeH2p+vwOTpXeAcyryABnN3X7ToHptqBavwV94KxVdGKA0A 4cC2vvmFodArrGyyOJTFaecC/kpxtDbkYxZdcxq1rMAd7K+2DlQP3kZtWwRrkjRW5OGM x1kiyEPtgEAN+ciNbt0PTL/hGI2jShXcBHIeCgVJFJ57wCSGp8XOzAtcoKmcaqdyLPZ2 RGOw== X-Gm-Message-State: AOAM533g8/fuXokZ/HGogY73SkWAiTrdOq0yE+BnvzKiunEXpLQXYIy2 JtOSWWE6aj3xWoPHlzjmwtuxsh+ZiMT9Ig== X-Received: by 2002:a17:90b:14cc:: with SMTP id jz12mr3382745pjb.210.1622193955215; Fri, 28 May 2021 02:25:55 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:54 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:49 +0530 Message-Id: <20210528092454.31874-5-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 05/10] lavfi/dnn: Fill Task using Common Function X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: GjTfbQ6PNs9a This commit adds a common function for filling the TaskItems in all three backends. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_common.c | 20 ++++++++++++++++++++ libavfilter/dnn/dnn_backend_common.h | 15 +++++++++++++++ libavfilter/dnn/dnn_backend_openvino.c | 23 +++++++---------------- 3 files changed, 42 insertions(+), 16 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index a522ab5650..4d9d3f79b1 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -49,3 +49,23 @@ int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func return 0; } + +DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc) { + if (task == NULL || exec_params == NULL || backend_model == NULL) + return DNN_ERROR; + if (do_ioproc != 0 && do_ioproc != 1) + return DNN_ERROR; + if (async != 0 && async != 1) + return DNN_ERROR; + + task->do_ioproc = do_ioproc; + task->async = async; + task->input_name = exec_params->input_name; + task->in_frame = exec_params->in_frame; + task->out_frame = exec_params->out_frame; + task->model = backend_model; + task->nb_output = exec_params->nb_output; + task->output_names = exec_params->output_names; + + return DNN_SUCCESS; +} diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index d962312c16..df59615f40 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -48,4 +48,19 @@ typedef struct InferenceItem { int ff_check_exec_params(void *ctx, DNNBackendType backend, DNNFunctionType func_type, DNNExecBaseParams *exec_params); +/** + * Fill the Task for Backend Execution. It should be called after + * checking execution parameters using ff_check_exec_params. + * + * @param task pointer to the allocated task + * @param exec_param pointer to execution parameters + * @param backend_model void pointer to the backend model + * @param async flag for async execution. Must be 0 or 1 + * @param do_ioproc flag for IO processing. Must be 0 or 1 + * + * @retval DNN_SUCCESS if successful + * @retval DNN_ERROR if flags are invalid or any parameter is NULL + */ +DNNReturnType ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backend_model, int async, int do_ioproc); + #endif diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index c2487c35be..709a772a4d 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -793,14 +793,9 @@ DNNReturnType ff_dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams * } } - task.do_ioproc = 1; - task.async = 0; - task.input_name = exec_params->input_name; - task.in_frame = exec_params->in_frame; - task.output_names = &exec_params->output_names[0]; - task.out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task.nb_output = exec_params->nb_output; - task.model = ov_model; + if (ff_dnn_fill_task(&task, exec_params, ov_model, 0, 1) != DNN_SUCCESS) { + return DNN_ERROR; + } if (extract_inference_from_task(ov_model->model->func_type, &task, ov_model->inference_queue, exec_params) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); @@ -841,14 +836,10 @@ DNNReturnType ff_dnn_execute_model_async_ov(const DNNModel *model, DNNExecBasePa return DNN_ERROR; } - task->do_ioproc = 1; - task->async = 1; - task->input_name = exec_params->input_name; - task->in_frame = exec_params->in_frame; - task->output_names = &exec_params->output_names[0]; - task->out_frame = exec_params->out_frame ? exec_params->out_frame : exec_params->in_frame; - task->nb_output = exec_params->nb_output; - task->model = ov_model; + if (ff_dnn_fill_task(task, exec_params, ov_model, 1, 1) != DNN_SUCCESS) { + return DNN_ERROR; + } + if (ff_queue_push_back(ov_model->task_queue, task) < 0) { av_freep(&task); av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); From patchwork Fri May 28 09:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27967 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368507iof; Fri, 28 May 2021 02:26:25 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwLBYIz5eDco4BRxC3jaB8nI+Q3p/DUyRIqXxFazQBrJAUK7wbjlLzLKrl+njbh5KQ39ycL X-Received: by 2002:a50:c94a:: with SMTP id p10mr8824390edh.115.1622193985122; Fri, 28 May 2021 02:26:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622193985; cv=none; d=google.com; s=arc-20160816; b=S7AHpfdqOKjihFgU0ffjIWBTpCo426wMNCihKHq5nDW+wkvWY7UhLHIdYR47cvJzIh mIQ70O0f4KiWr5SRs+XVg4Kgao+cvgUB91HQ6uWnyjjuxL28W58L+h3Yfz7M2SXkbKZZ KBEmltJ+QusfByLjPwTbKYdEs1D0KtOl1Nmk8abjNPnT9xZCd2m45GYTvh5+dhLWF/NO 4zIU3F4IW74S6b/fUK06s0+0A3oF1Yz09BNqVE7YleJ34KX8LFdnuLcnqe6XrH2fFcZ7 ZWTr6n1jPGTcleKiUGNq78pAhuaszdpIqez3n+1GYV5wfMDquiKKOFR0UjJPpDu5oAH/ 9qpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=V8Q4Ap5Lwj6xaABXzHnhHgkI9n27UWRQMbzUy/WA7R0=; b=ynr96h9NtFGjU8tdo9KsriTcfOxw5aF/grROgUwYLAUpVn2DXZnRXSEK/OkgP0WC24 MpOAb62nbDHA7LLNZxFyUNimWcIcgdgn98tf8qsLIKYZF3cfnQF+HcywnmM0qmsdPHhX nkX8+kddQFS5nlmP1ynSC2vPUExgDXdc7DHeR7+SrN4/1pQvlbEAdp68afDoYvJ+L4SB Qdt/8OzZ6y0VU17TgIJlilLXcHzE/ilZACL9TOPsPRS3z1vrncaIlkUOhL1ib9QCa6tT bwdIn0K8EGzhqD84TfD28rPoZFNutLrsiq6dpAWaXOtO1eqxQ2ha9e6jJj/qT6Sfs8Co 9FwA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WKrMgMts; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id x12si4483943ejj.664.2021.05.28.02.26.24; Fri, 28 May 2021 02:26:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=WKrMgMts; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 25574689F42; Fri, 28 May 2021 12:26:05 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 00F85680B73 for ; Fri, 28 May 2021 12:25:58 +0300 (EEST) Received: by mail-pj1-f52.google.com with SMTP id gb21-20020a17090b0615b029015d1a863a91so4137414pjb.2 for ; Fri, 28 May 2021 02:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7z9zmYiB7Vykvc5goqbWnuQbyq/aQLALXYJdujDS/lk=; b=WKrMgMts3w40mRegaTixOxNIVsMrptX581smDItAVz+InGwjLfxjfjRgpA92kTxxyV Jn9FMNCM+/OjOW5lnv8ZDPzEx8xpMHOCs/0zhNrU3xSuu+PvT95UPdXvLCLY2TlH+Q6T QFhSweVQym1/t4emum4ivgJFqiaVJpnq+tHEqRSgxq8B76XWUZmzSWDnidr2ZA+mcd7C Q/Vih9lARV1OIywbakCcAeSB7QZAlUlchTEV/MK3vDIzz0u3XEsC2gaYGppBM238Rz6r i5KprsSXUMjsh6dM5V0SPkeFgCXRC+EHuXKUiAyTpLpWq4yHJEy/lfRQkSoecZnahCFZ o2dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7z9zmYiB7Vykvc5goqbWnuQbyq/aQLALXYJdujDS/lk=; b=XuUzNdn6PkZAZciTuRJaex+YiRlnlXa9nQRiPbGtLadkLW56C4HrbWJyyiRj1j7711 rDNc8gotjHFafdD4muQfdSjI8OYUzsdgBs1XXEA3QYhXvf6PxG9i1d1FM+wE5nV3h+0e euIEfgkdUwltR/LVaZuQ7rJw5VwZ04nKPCQgEFJv+e9SVSoYNFpYVwAm79YwQHj9UYgB ZBnTq0oMZ/turLfTV56E6aPSAXEMKTM9uSY4KOkKFACUZPFL5H32y8QZeDtZVQjXD/il iJPROK4MHzMROnKl+rkn0W7MU4JwRETjSbhS9IisiBoHmyUpWZnOR+3ZjY0JVPdF9vAe n7dQ== X-Gm-Message-State: AOAM530XPp5z4p3u3jPwxLHezcqOuXa/xQNnfFQ+zGE79f90EyZNv5ic tEeNXSHWvABLLlv6sKpy7Frm2QBzhTtbIA== X-Received: by 2002:a17:90a:7342:: with SMTP id j2mr3526497pjs.10.1622193957036; Fri, 28 May 2021 02:25:57 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:56 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:50 +0530 Message-Id: <20210528092454.31874-6-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 06/10] lavfi/dnn_backend_tf: Request-based Execution X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: kkURNxTk1Vag This commit adds RequestItem and rearranges the existing sync execution mechanism to use request-based execution. It will help in adding async functionality to the TensorFlow backend later. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 297 +++++++++++++++++++++---------- 1 file changed, 206 insertions(+), 91 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 4c16c2bdb0..793b108e55 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -35,10 +35,13 @@ #include "dnn_backend_native_layer_maximum.h" #include "dnn_io_proc.h" #include "dnn_backend_common.h" +#include "safe_queue.h" +#include "queue.h" #include typedef struct TFOptions{ char *sess_config; + uint32_t nireq; } TFOptions; typedef struct TFContext { @@ -52,26 +55,79 @@ typedef struct TFModel{ TF_Graph *graph; TF_Session *session; TF_Status *status; + SafeQueue *request_queue; + Queue *inference_queue; } TFModel; +typedef struct tf_infer_request { + TF_Output *tf_outputs; + TF_Tensor **output_tensors; + TF_Output *tf_input; + TF_Tensor *input_tensor; +} tf_infer_request; + +typedef struct RequestItem { + tf_infer_request *infer_request; + InferenceItem *inference; + // further properties will be added later for async +} RequestItem; + #define OFFSET(x) offsetof(TFContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM static const AVOption dnn_tensorflow_options[] = { { "sess_config", "config for SessionOptions", OFFSET(options.sess_config), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, + { "nireq", "number of request", OFFSET(options.nireq), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, { NULL } }; AVFILTER_DEFINE_CLASS(dnn_tensorflow); -static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_name, AVFrame *in_frame, - const char **output_names, uint32_t nb_output, AVFrame *out_frame, - int do_ioproc); +static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue); static void free_buffer(void *data, size_t length) { av_freep(&data); } +static void tf_free_request(tf_infer_request *request) +{ + if (!request) + return; + if (request->input_tensor) { + TF_DeleteTensor(request->input_tensor); + request->input_tensor = NULL; + } + av_freep(&request->tf_input); + av_freep(&request->tf_outputs); + av_freep(&request->output_tensors); +} + +static tf_infer_request* tf_create_inference_request(void) +{ + tf_infer_request* infer_request = av_malloc(sizeof(tf_infer_request)); + infer_request->tf_outputs = NULL; + infer_request->tf_input = NULL; + infer_request->input_tensor = NULL; + infer_request->output_tensors = NULL; + return infer_request; +} + +static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) +{ + InferenceItem *inference = av_malloc(sizeof(*inference)); + if (!inference) { + return DNN_ERROR; + } + task->inference_todo = 1; + task->inference_done = 0; + inference->task = task; + if (ff_queue_push_back(inference_queue, inference) < 0) { + av_freep(&inference); + return DNN_ERROR; + } + return DNN_SUCCESS; +} + static TF_Buffer *read_graph(const char *model_filename) { TF_Buffer *graph_buf; @@ -171,6 +227,8 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu TFContext *ctx = &tf_model->ctx; AVFrame *in_frame = av_frame_alloc(); AVFrame *out_frame = NULL; + TaskItem task; + RequestItem *request; if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); @@ -187,7 +245,27 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu in_frame->width = input_width; in_frame->height = input_height; - ret = execute_model_tf(tf_model->model, input_name, in_frame, &output_name, 1, out_frame, 0); + task.do_ioproc = 0; + task.async = 0; + task.input_name = input_name; + task.in_frame = in_frame; + task.output_names = &output_name; + task.out_frame = out_frame; + task.model = tf_model; + task.nb_output = 1; + + if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = execute_model_tf(request, tf_model->inference_queue); *output_width = out_frame->width; *output_height = out_frame->height; @@ -691,6 +769,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ { DNNModel *model = NULL; TFModel *tf_model = NULL; + TFContext *ctx = NULL; model = av_mallocz(sizeof(DNNModel)); if (!model){ @@ -704,10 +783,11 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } tf_model->ctx.class = &dnn_tensorflow_class; tf_model->model = model; + ctx = &tf_model->ctx; //parse options - av_opt_set_defaults(&tf_model->ctx); - if (av_opt_set_from_string(&tf_model->ctx, options, NULL, "=", "&") < 0) { + av_opt_set_defaults(&ctx); + if (av_opt_set_from_string(&ctx, options, NULL, "=", "&") < 0) { av_log(&tf_model->ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); av_freep(&tf_model); av_freep(&model); @@ -723,6 +803,19 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } } + if (ctx->options.nireq <= 0) { + ctx->options.nireq = av_cpu_count() / 2 + 1; + } + + tf_model->request_queue = ff_safe_queue_create(); + + for (int i = 0; i < ctx->options.nireq; i++) { + RequestItem *item = av_mallocz(sizeof(*item)); + item->infer_request = tf_create_inference_request(); + ff_safe_queue_push_back(tf_model->request_queue, item); + } + + tf_model->inference_queue = ff_queue_create(); model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -733,168 +826,176 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(const DNNModel *model, const char *input_name, AVFrame *in_frame, - const char **output_names, uint32_t nb_output, AVFrame *out_frame, - int do_ioproc) +static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) { - TF_Output *tf_outputs; - TFModel *tf_model = model->model; - TFContext *ctx = &tf_model->ctx; + TFModel *tf_model; + TFContext *ctx; + tf_infer_request *infer_request; + InferenceItem *inference; + TaskItem *task; DNNData input, *outputs; - TF_Tensor **output_tensors; - TF_Output tf_input; - TF_Tensor *input_tensor; - if (get_input_tf(tf_model, &input, input_name) != DNN_SUCCESS) + inference = ff_queue_pop_front(inference_queue); + av_assert0(inference); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + request->inference = inference; + + if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) return DNN_ERROR; - input.height = in_frame->height; - input.width = in_frame->width; - tf_input.oper = TF_GraphOperationByName(tf_model->graph, input_name); - if (!tf_input.oper){ + infer_request = request->infer_request; + input.height = task->in_frame->height; + input.width = task->in_frame->width; + + infer_request->tf_input = av_malloc(sizeof(TF_Output)); + infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); + if (!infer_request->tf_input->oper){ av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name); return DNN_ERROR; } - tf_input.index = 0; - input_tensor = allocate_input_tensor(&input); - if (!input_tensor){ + infer_request->tf_input->index = 0; + infer_request->input_tensor = allocate_input_tensor(&input); + if (!infer_request->input_tensor){ av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); return DNN_ERROR; } - input.data = (float *)TF_TensorData(input_tensor); + input.data = (float *)TF_TensorData(infer_request->input_tensor); switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: - if (do_ioproc) { + if (task->do_ioproc) { if (tf_model->model->frame_pre_proc != NULL) { - tf_model->model->frame_pre_proc(in_frame, &input, tf_model->model->filter_ctx); + tf_model->model->frame_pre_proc(task->in_frame, &input, tf_model->model->filter_ctx); } else { - ff_proc_from_frame_to_dnn(in_frame, &input, ctx); + ff_proc_from_frame_to_dnn(task->in_frame, &input, ctx); } } break; case DFT_ANALYTICS_DETECT: - ff_frame_to_dnn_detect(in_frame, &input, ctx); + ff_frame_to_dnn_detect(task->in_frame, &input, ctx); break; default: avpriv_report_missing_feature(ctx, "model function type %d", tf_model->model->func_type); break; } - tf_outputs = av_malloc_array(nb_output, sizeof(*tf_outputs)); - if (tf_outputs == NULL) { - TF_DeleteTensor(input_tensor); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); \ + infer_request->tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); + if (infer_request->tf_outputs == NULL) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); return DNN_ERROR; } - output_tensors = av_mallocz_array(nb_output, sizeof(*output_tensors)); - if (!output_tensors) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); \ + infer_request->output_tensors = av_mallocz_array(task->nb_output, sizeof(*infer_request->output_tensors)); + if (!infer_request->output_tensors) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); return DNN_ERROR; } - for (int i = 0; i < nb_output; ++i) { - tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, output_names[i]); - if (!tf_outputs[i].oper) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", output_names[i]); \ + for (int i = 0; i < task->nb_output; ++i) { + infer_request->tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); + if (!infer_request->tf_outputs[i].oper) { + av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); return DNN_ERROR; } - tf_outputs[i].index = 0; + infer_request->tf_outputs[i].index = 0; } TF_SessionRun(tf_model->session, NULL, - &tf_input, &input_tensor, 1, - tf_outputs, output_tensors, nb_output, - NULL, 0, NULL, tf_model->status); + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); if (TF_GetCode(tf_model->status) != TF_OK) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; } - outputs = av_malloc_array(nb_output, sizeof(*outputs)); + outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { - TF_DeleteTensor(input_tensor); - av_freep(&tf_outputs); - av_freep(&output_tensors); - av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); \ + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); return DNN_ERROR; } - for (uint32_t i = 0; i < nb_output; ++i) { - outputs[i].height = TF_Dim(output_tensors[i], 1); - outputs[i].width = TF_Dim(output_tensors[i], 2); - outputs[i].channels = TF_Dim(output_tensors[i], 3); - outputs[i].data = TF_TensorData(output_tensors[i]); - outputs[i].dt = TF_TensorType(output_tensors[i]); + for (uint32_t i = 0; i < task->nb_output; ++i) { + outputs[i].height = TF_Dim(infer_request->output_tensors[i], 1); + outputs[i].width = TF_Dim(infer_request->output_tensors[i], 2); + outputs[i].channels = TF_Dim(infer_request->output_tensors[i], 3); + outputs[i].data = TF_TensorData(infer_request->output_tensors[i]); + outputs[i].dt = TF_TensorType(infer_request->output_tensors[i]); } - switch (model->func_type) { + switch (tf_model->model->func_type) { case DFT_PROCESS_FRAME: //it only support 1 output if it's frame in & frame out - if (do_ioproc) { + if (task->do_ioproc) { if (tf_model->model->frame_post_proc != NULL) { - tf_model->model->frame_post_proc(out_frame, outputs, tf_model->model->filter_ctx); + tf_model->model->frame_post_proc(task->out_frame, outputs, tf_model->model->filter_ctx); } else { - ff_proc_from_dnn_to_frame(out_frame, outputs, ctx); + ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } } else { - out_frame->width = outputs[0].width; - out_frame->height = outputs[0].height; + task->out_frame->width = outputs[0].width; + task->out_frame->height = outputs[0].height; } break; case DFT_ANALYTICS_DETECT: - if (!model->detect_post_proc) { + if (!tf_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return DNN_ERROR; } - model->detect_post_proc(out_frame, outputs, nb_output, model->filter_ctx); + tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; default: - for (uint32_t i = 0; i < nb_output; ++i) { - if (output_tensors[i]) { - TF_DeleteTensor(output_tensors[i]); + for (uint32_t i = 0; i < task->nb_output; ++i) { + if (infer_request->output_tensors[i]) { + TF_DeleteTensor(infer_request->output_tensors[i]); } } - TF_DeleteTensor(input_tensor); - av_freep(&output_tensors); - av_freep(&tf_outputs); - av_freep(&outputs); - av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); return DNN_ERROR; } - - for (uint32_t i = 0; i < nb_output; ++i) { - if (output_tensors[i]) { - TF_DeleteTensor(output_tensors[i]); + for (uint32_t i = 0; i < task->nb_output; ++i) { + if (infer_request->output_tensors[i]) { + TF_DeleteTensor(infer_request->output_tensors[i]); } } - TF_DeleteTensor(input_tensor); - av_freep(&output_tensors); - av_freep(&tf_outputs); + task->inference_done++; + tf_free_request(infer_request); av_freep(&outputs); - return DNN_SUCCESS; + ff_safe_queue_push_back(tf_model->request_queue, request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { TFModel *tf_model = model->model; TFContext *ctx = &tf_model->ctx; + TaskItem task; + RequestItem *request; if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { - return DNN_ERROR; + return DNN_ERROR; + } + + if (ff_dnn_fill_task(&task, exec_params, tf_model, 0, 1) != DNN_SUCCESS) { + return DNN_ERROR; + } + + if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; } - return execute_model_tf(model, exec_params->input_name, exec_params->in_frame, - exec_params->output_names, exec_params->nb_output, exec_params->out_frame, 1); + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + return execute_model_tf(request, tf_model->inference_queue); } void ff_dnn_free_model_tf(DNNModel **model) @@ -903,6 +1004,20 @@ void ff_dnn_free_model_tf(DNNModel **model) if (*model){ tf_model = (*model)->model; + while (ff_safe_queue_size(tf_model->request_queue) != 0) { + RequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); + tf_free_request(item->infer_request); + av_freep(&item->infer_request); + av_freep(&item); + } + ff_safe_queue_destroy(tf_model->request_queue); + + while (ff_queue_size(tf_model->inference_queue) != 0) { + InferenceItem *item = ff_queue_pop_front(tf_model->inference_queue); + av_freep(&item); + } + ff_queue_destroy(tf_model->inference_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } From patchwork Fri May 28 09:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27962 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368682iof; Fri, 28 May 2021 02:26:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzJx3eYpYdPsZE/qj2rMV0Ixwb8NKRuPNiQSyy7EdynGP0+QejkWtWQcUqsep5/c7azaNtp X-Received: by 2002:a17:906:8303:: with SMTP id j3mr5341550ejx.403.1622194009198; Fri, 28 May 2021 02:26:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622194009; cv=none; d=google.com; s=arc-20160816; b=xOKQyaWvhR7H5cQn64aekILskpWMoz2RPIu3q16lHDz6XEaecx5FJKfGEtrJa5oqtm RYyIYOgLtncBXX2xPKY1Vbg9aQd/pJrhFMkYgD7Mr7+gA9hqidikBcwTHFq+SwzjUA5L txm8SP1GAKdS/Fld/1+9KPXtWDenKac9Dp2h7Z6KfSEQvSeTuzCk3sQQxV1s1vhFXMFx Im1jySC93Xp3MaTJEVnaBM74IfG14babQq1wBnjoV8/vVAT3oHfGX+vKpodi8BHeD7w4 5VbXikfZq0FUl7K34zr7mBi3Gnp2cSQwDQJBUEYo9Xq7odhSaiZaL0s0bUskAXhH0+Ya mcOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=H5EOkytwAUDYgiYxYe0DDBcOY/67AscbbghqeOLsPZM=; b=PA+NVcb9ygN3i0pv8LBvejgOPy7+0hKlUlwAWaJvnBrngeSqaOL2TJrm6oMlaHykCf OPBdEdr6iECUUPJW0UZXQS0u/S9v7903BPymnzXJeFJCAVBi48SNCDW9lU80SkETj1Q8 gKD9E0+J8lvaI3YUZR8p5JeT+0s3f28bmu8MbeT2+S3akrjSzqfnKIamgO8ee/pp1Fn2 tddRBZCOXg1Sh6M/gkZz7so05vfXA3dYe5IkS1fvN5p2Us+DQRyRnmZ7Z7TOpa6TUxg2 2vLQuaFc/dirkVTlZm3Qr6qennPPxAUFGGTZZuOVKFJyExOkfnJs4/+5GUn3FNA8tN5V iWWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=lpSJ+LBz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id he12si4464831ejc.636.2021.05.28.02.26.48; Fri, 28 May 2021 02:26:49 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=lpSJ+LBz; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 054BD68A08E; Fri, 28 May 2021 12:26:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C074A689EA6 for ; Fri, 28 May 2021 12:26:00 +0300 (EEST) Received: by mail-pj1-f42.google.com with SMTP id f3-20020a17090a4a83b02901619627235bso3344pjh.1 for ; Fri, 28 May 2021 02:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uHkRtYZa2QhT8vt5BjboRgyhSUN5OXGfLaCclbnxZE0=; b=lpSJ+LBzwL1UAXW7MYTPyzDX5bdCOSkcCKpAgbFO/h0dUvUswhR8v6+MnT1EFd2XjK 1jPt58qvLiRBYAHIFneoc/H16MZmrqME/uHU4m5hdhhkqYjyPdAh59/PQb6NrlO2yhas CbhlTq0slIBRiI9GDM3BOOdkkjF/MMImO5092i+FF1skfWEmfnE+Chalaq/Q2t0Gx0xV 5nZUz3WUSy8rK52Lykh3RMZDEiIe51GUZXGc07RByc6BRzI1b7mehrVlkjFh5qR5wFTG Vh5nSULmhJsh6S2PvEdSmhGsFOBDF1EQ0+MYbxCrGUvV41ZklNGnQZSPXXFd0qZA+RZN K0IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uHkRtYZa2QhT8vt5BjboRgyhSUN5OXGfLaCclbnxZE0=; b=kX4+XVpUHwz1teHILUsYwawLJ4uDtLWJizSFiJJ0s3EtfJ/ikETa1Q5jqiV+u7LTlx H9L+BClPRHQq1UM0TgNeirXQLs6JI8nrtK8RcmfVK4tURzNpzDJwxuFFLDbSnKvkgval 0zBnjvLNmpoVEIaEYWGoYTA37PrFoaRDUT2aqSzOFaJ3XrLfy21ktKDl/KK84qHJaGJC WmziwFVhVl/bFTHwSEtBhQ3fdvsVxB86lhwLkY6ZQksJz8p/K5tBttcq44rfhWrauy/+ kmf6prTrqU8GOsONbvWPdfNbavWgyarCmKWLGfVL0xBIEKMq/BX8IOkAr7bHdWhdelw9 3naA== X-Gm-Message-State: AOAM533VzAIGBxbXkPicvZlcmS9f0wrD/OhbDj+c6/tYQfup+jnYx8mL PKMe7lq0NPoYqGR8DV6qZhnZRuoYbZFcNA== X-Received: by 2002:a17:902:724c:b029:ef:571f:8894 with SMTP id c12-20020a170902724cb02900ef571f8894mr7222607pll.49.1622193958981; Fri, 28 May 2021 02:25:58 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:25:58 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:51 +0530 Message-Id: <20210528092454.31874-7-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 07/10] lavfi/dnn_backend_tf: Separate function for filling RequestItem and callback X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: Tk9CsIegggka This commit rearranges the existing code to create two separate functions for filling request with execution data and the completion callback. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 81 ++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 24 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 793b108e55..5d34da5db1 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -826,20 +826,16 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ return model; } -static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) -{ - TFModel *tf_model; - TFContext *ctx; - tf_infer_request *infer_request; +static DNNReturnType fill_model_input_tf(TFModel *tf_model, RequestItem *request) { + DNNData input; InferenceItem *inference; TaskItem *task; - DNNData input, *outputs; + tf_infer_request *infer_request; + TFContext *ctx = &tf_model->ctx; - inference = ff_queue_pop_front(inference_queue); + inference = ff_queue_pop_front(tf_model->inference_queue); av_assert0(inference); task = inference->task; - tf_model = task->model; - ctx = &tf_model->ctx; request->inference = inference; if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) @@ -852,7 +848,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que infer_request->tf_input = av_malloc(sizeof(TF_Output)); infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); if (!infer_request->tf_input->oper){ - av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", input_name); + av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); return DNN_ERROR; } infer_request->tf_input->index = 0; @@ -902,22 +898,23 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que infer_request->tf_outputs[i].index = 0; } - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); - if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); - av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); - return DNN_ERROR; - } + return DNN_SUCCESS; +} + +static void infer_completion_callback(void *args) { + RequestItem *request = args; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + DNNData *outputs; + tf_infer_request *infer_request = request->infer_request; + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { tf_free_request(infer_request); av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); - return DNN_ERROR; + return; } for (uint32_t i = 0; i < task->nb_output; ++i) { @@ -944,7 +941,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que case DFT_ANALYTICS_DETECT: if (!tf_model->model->detect_post_proc) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); - return DNN_ERROR; + return; } tf_model->model->detect_post_proc(task->out_frame, outputs, task->nb_output, tf_model->model->filter_ctx); break; @@ -955,7 +952,7 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que } } av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); - return DNN_ERROR; + return; } for (uint32_t i = 0; i < task->nb_output; ++i) { if (infer_request->output_tensors[i]) { @@ -966,7 +963,43 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que tf_free_request(infer_request); av_freep(&outputs); ff_safe_queue_push_back(tf_model->request_queue, request); - return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; +} + +static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) +{ + TFModel *tf_model; + TFContext *ctx; + tf_infer_request *infer_request; + InferenceItem *inference; + TaskItem *task; + + inference = ff_queue_peek_front(inference_queue); + task = inference->task; + tf_model = task->model; + ctx = &tf_model->ctx; + + if (task->async) { + avpriv_report_missing_feature(ctx, "Async execution not supported"); + return DNN_ERROR; + } else { + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { + return DNN_ERROR; + } + + infer_request = request->infer_request; + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); + if (TF_GetCode(tf_model->status) != TF_OK) { + tf_free_request(infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; + } + infer_completion_callback(request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; + } } DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params) From patchwork Fri May 28 09:24:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27968 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368843iof; Fri, 28 May 2021 02:27:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwGMGeIZv0/cyXUO+irCdwC2BP/163YRUmcbZ90BVsbqxnCy86msTfEoaR5iffmZOu7Awl4 X-Received: by 2002:a17:906:2bd3:: with SMTP id n19mr8057550ejg.210.1622194029998; Fri, 28 May 2021 02:27:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622194029; cv=none; d=google.com; s=arc-20160816; b=iJosB/pBhjVpcW8WpAdisY8fMcMAGxUYFju64v4YBohhGEjN50Ea7iZNaNPCHkSZvT YcTIGskbWd8HM94MjXLlAf+bZstWXq2Ytl1UmX7xjqMYnFDWyN32ftf6R0Uh56QLqzxE 0rkZRZL0VpSlZTosyyfNfq9RrGJnni4QRJ6+numbnfV8pGUtsmstA4aVeMxbsPLligj5 5l072RUkGwZyCTes4VW/GqV9Bp6DTozRjnFsTQ+EZd9PmMtB0s/V2R2AcxBreLxnwOst tWLnaNZFM+gUOF5rq1uX9ag02W4c0L5/igUIAg9ylVS4/Mfm9iTlIeWLKWqYi8lvs3Ne oRyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=y9tm2CC+HH57S/b7kSf7gLEJa7rMEMwGDWqg8Ja6S1Y=; b=KTSFJrL8FinVkU3zJEhVruJNlnuH8Li1E7CB/fg2dWcrYuHK1VMbksVD5Tukm1tdK0 wTjrJmznS9EUftONj+MPR9chXZ4ZX+3CNZFNcxrXgxVKeGGd8eVFR54RlEorzXFOmrgm hyI7pa3C/o0SN1JzjgVF/X/6nEH57gqu1H2JRCKhAGLNzgA1eFyHrXTfM0JfoBET9P/k A511Ehda6JU5XgOMRiuOo3GNu26eOSL/ZGlRMT8E9GGZKF3h/zr10fx2YwZ4w3ooez5J 9/NSjhUyrlDr/hnw6t+w+xPXgFr4SjVUFaHYF5M9ozclIkl1ge2aue2PbqbgeyXepDh9 f5UA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=FxqIdv3T; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id t10si3122567ejs.653.2021.05.28.02.27.09; Fri, 28 May 2021 02:27:09 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=FxqIdv3T; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E02B568A000; Fri, 28 May 2021 12:26:10 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CB0A8689F7D for ; Fri, 28 May 2021 12:26:02 +0300 (EEST) Received: by mail-pg1-f182.google.com with SMTP id q15so2064368pgg.12 for ; Fri, 28 May 2021 02:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cF+218VUfGoGEnuqUvTirbIosUCOEyi9IQHG6D+3bp4=; b=FxqIdv3TjVFkcV26s4Ww8KY7EQlAqRkJqEN7/nvhCERJ4XiCuae+mCqcov7WRaGA9z hFc4XYx8DuYoc8/rmX4hUdAG1jmvrpVCcAz+ITLPrNkD1ldXrfGwPOOTzDwMcRTzZ49M XWOFode0EO8eomoeuOarILHj+c90+nmvI7E9SZ4Oq8EtxpxpGfOdlKzAr1uOtVtA9mHF FXlwNm89KZmphNQ9rLo3Xayydbq/tBlJ4X/LbA+CHx5pZtHPP0XVq6Hqsl83RPLP3i7b aQsSGcJazLEWVO5i1A4Ue1vIF34rVyuk1m4m7cuJwXUX5PKuepLbSNjDpHEUQjKhoglY VFHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cF+218VUfGoGEnuqUvTirbIosUCOEyi9IQHG6D+3bp4=; b=BHNequxtHc2m6DL2ALQ+NeeD33vD9DSBllIZi66huTjgaT2Q9aOY9MhsxMSE7khwNa +Dlv34lprgNbYJhK/T42cT/GURXgGhBvVjvUBbS76r1mPWVqwsiADMtbUqsRJKwg8D8s wzO6KslzrpcbvTOWQfHMkbwi7YqYI1mUIxCq33UrYh7y3d+GUiuuPDGCYa8G4voFUIHR wc76Pxm7SVzn3tIQCXCYJrFNCyItUOUHhCdjQTlXohDkaluySxqElpB08P4QjRCO7RBT yi07zzv1kdQFW2AleK9iuQhRIDnhAO0Rc4aCAU063NNm23qSr2dbaaelTs6T6JMUBGhQ 7Xww== X-Gm-Message-State: AOAM533nTVxJmvx/JSZaHyJ1F+6c8FBrR0zaKLRRYG/KQZsLrsJ3safH 6meQ4w0oIJJ/hRsg794d5OMyykfbgcohow== X-Received: by 2002:a63:5d66:: with SMTP id o38mr8128871pgm.444.1622193960976; Fri, 28 May 2021 02:26:00 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.25.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:26:00 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:52 +0530 Message-Id: <20210528092454.31874-8-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 08/10] lavfi/dnn_backend_tf: Error Handling X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: WlciRI0TAamv This commit adds handling for cases where an error may occur, clearing the allocated memory resources. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 100 +++++++++++++++++++++++-------- 1 file changed, 74 insertions(+), 26 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 5d34da5db1..31746deef4 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -114,14 +114,18 @@ static tf_infer_request* tf_create_inference_request(void) static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; InferenceItem *inference = av_malloc(sizeof(*inference)); if (!inference) { + av_log(ctx, AV_LOG_ERROR, "Unable to allocate space for InferenceItem\n"); return DNN_ERROR; } task->inference_todo = 1; task->inference_done = 0; inference->task = task; if (ff_queue_push_back(inference_queue, inference) < 0) { + av_log(ctx, AV_LOG_ERROR, "Failed to push back inference_queue.\n"); av_freep(&inference); return DNN_ERROR; } @@ -232,14 +236,15 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu if (!in_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input frame\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto final; } out_frame = av_frame_alloc(); if (!out_frame) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output frame\n"); - av_frame_free(&in_frame); - return DNN_ERROR; + ret = DNN_ERROR; + goto final; } in_frame->width = input_width; @@ -256,19 +261,22 @@ static DNNReturnType get_output_tf(void *model, const char *input_name, int inpu if (extract_inference_from_task(&task, tf_model->inference_queue) != DNN_SUCCESS) { av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto final; } request = ff_safe_queue_pop_front(tf_model->request_queue); if (!request) { av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); - return DNN_ERROR; + ret = DNN_ERROR; + goto final; } ret = execute_model_tf(request, tf_model->inference_queue); *output_width = out_frame->width; *output_height = out_frame->height; +final: av_frame_free(&out_frame); av_frame_free(&in_frame); return ret; @@ -788,18 +796,13 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ //parse options av_opt_set_defaults(&ctx); if (av_opt_set_from_string(&ctx, options, NULL, "=", "&") < 0) { - av_log(&tf_model->ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); - av_freep(&tf_model); - av_freep(&model); - return NULL; + av_log(&ctx, AV_LOG_ERROR, "Failed to parse options \"%s\"\n", options); + goto err; } if (load_tf_model(tf_model, model_filename) != DNN_SUCCESS){ if (load_native_model(tf_model, model_filename) != DNN_SUCCESS){ - av_freep(&tf_model); - av_freep(&model); - - return NULL; + goto err; } } @@ -808,14 +811,34 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ } tf_model->request_queue = ff_safe_queue_create(); + if (!tf_model->request_queue) { + goto err; + } for (int i = 0; i < ctx->options.nireq; i++) { RequestItem *item = av_mallocz(sizeof(*item)); + if (!item) { + goto err; + } item->infer_request = tf_create_inference_request(); - ff_safe_queue_push_back(tf_model->request_queue, item); + if (!item->infer_request) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for TensorFlow inference request\n"); + av_freep(&item); + goto err; + } + + if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { + av_freep(&item->infer_request); + av_freep(&item); + goto err; + } } tf_model->inference_queue = ff_queue_create(); + if (!tf_model->inference_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -824,6 +847,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ model->func_type = func_type; return model; +err: + ff_dnn_free_model_tf(&model); + return NULL; } static DNNReturnType fill_model_input_tf(TFModel *tf_model, RequestItem *request) { @@ -838,24 +864,31 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, RequestItem *request task = inference->task; request->inference = inference; - if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) - return DNN_ERROR; + if (get_input_tf(tf_model, &input, task->input_name) != DNN_SUCCESS) { + goto err; + } infer_request = request->infer_request; input.height = task->in_frame->height; input.width = task->in_frame->width; infer_request->tf_input = av_malloc(sizeof(TF_Output)); + if (!infer_request->tf_input) { + av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); + goto err; + } + infer_request->tf_input->oper = TF_GraphOperationByName(tf_model->graph, task->input_name); if (!infer_request->tf_input->oper){ av_log(ctx, AV_LOG_ERROR, "Could not find \"%s\" in model\n", task->input_name); - return DNN_ERROR; + goto err; } infer_request->tf_input->index = 0; + infer_request->input_tensor = allocate_input_tensor(&input); if (!infer_request->input_tensor){ av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for input tensor\n"); - return DNN_ERROR; + goto err; } input.data = (float *)TF_TensorData(infer_request->input_tensor); @@ -880,27 +913,35 @@ static DNNReturnType fill_model_input_tf(TFModel *tf_model, RequestItem *request infer_request->tf_outputs = av_malloc_array(task->nb_output, sizeof(TF_Output)); if (infer_request->tf_outputs == NULL) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *tf_outputs\n"); - return DNN_ERROR; + goto err; } infer_request->output_tensors = av_mallocz_array(task->nb_output, sizeof(*infer_request->output_tensors)); if (!infer_request->output_tensors) { av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for output tensor\n"); - return DNN_ERROR; + goto err; } - for (int i = 0; i < task->nb_output; ++i) { infer_request->tf_outputs[i].oper = TF_GraphOperationByName(tf_model->graph, task->output_names[i]); if (!infer_request->tf_outputs[i].oper) { av_log(ctx, AV_LOG_ERROR, "Could not find output \"%s\" in model\n", task->output_names[i]); - return DNN_ERROR; + goto err; } infer_request->tf_outputs[i].index = 0; } return DNN_SUCCESS; +err: + for (uint32_t i = 0; i < task->nb_output; ++i) { + if (infer_request->output_tensors[i]) { + TF_DeleteTensor(infer_request->output_tensors[i]); + } + } + tf_free_request(infer_request); + return DNN_ERROR; } + static void infer_completion_callback(void *args) { RequestItem *request = args; InferenceItem *inference = request->inference; @@ -912,9 +953,8 @@ static void infer_completion_callback(void *args) { outputs = av_malloc_array(task->nb_output, sizeof(*outputs)); if (!outputs) { - tf_free_request(infer_request); av_log(ctx, AV_LOG_ERROR, "Failed to allocate memory for *outputs\n"); - return; + goto final; } for (uint32_t i = 0; i < task->nb_output; ++i) { @@ -952,7 +992,7 @@ static void infer_completion_callback(void *args) { } } av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); - return; + goto final; } for (uint32_t i = 0; i < task->nb_output; ++i) { if (infer_request->output_tensors[i]) { @@ -960,9 +1000,13 @@ static void infer_completion_callback(void *args) { } } task->inference_done++; +final: tf_free_request(infer_request); av_freep(&outputs); - ff_safe_queue_push_back(tf_model->request_queue, request); + + if (ff_safe_queue_push_back(tf_model->request_queue, request) < 0) { + av_log(ctx, AV_LOG_ERROR, "Failed to push back request_queue.\n"); + } } static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) @@ -974,6 +1018,10 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que TaskItem *task; inference = ff_queue_peek_front(inference_queue); + if (!inference) { + av_log(NULL, AV_LOG_ERROR, "Failed to get inference item\n"); + return DNN_ERROR; + } task = inference->task; tf_model = task->model; ctx = &tf_model->ctx; From patchwork Fri May 28 09:24:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27964 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368592iof; Fri, 28 May 2021 02:26:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxg9Z2AZYFwsJW9HnSFMkVqe/QaOgiwV5NuUMdthX9M16TbE7e1jO5wyoSi5FaiRq0aPgxu X-Received: by 2002:a17:907:6289:: with SMTP id nd9mr8040507ejc.384.1622193997192; Fri, 28 May 2021 02:26:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622193997; cv=none; d=google.com; s=arc-20160816; b=nex3iORkUbWqfxO2DfF7x3ypXBGdmYOYmfXWrpMRlSA6unihDzZQql8+JLotVMev3v oGWUEpWMcZmlN2hV/MlYvMYE0ryOj+cUOyPvhkvF2p3t1J3WBFajMTqdTZWCxpNrb7Jx rIudh7KGsFpDiQ/io9A1SHeK2U20T5ON08RUzkBBn9h3IIdEj/YlVelFZkZAAHry0Scv lEhgVL1j4ooQvPe1SnIkK/2i3gEC3IgpV8V4FGz91XzRdCtYXl7t9mgAUhf8S1nfQs21 vIy5aBqIrLhFPEs32YrxNWYzQ+VWqz7KMSb8y+1Ep/pwYaJ0UBB3iu8/F9tOK+p7FmPT qB4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=LuN/6FBW4LbEdpXK+g+c7i/16EMdF3ElqmfB2Nmj2IU=; b=h0TiE2zKcWf3dG1z5yJ3imAO1Ll4xu8d8V440/+PkmbxbkycQtJjZp+vz1vDXjvs6A CP7gXeYZHv8BwgHBQ33hMhBivYqB0ti16aIqFkWJTBXWFmLdetjX64oh5NiyoKNQGjdt u9m0tJrN5Rxx3dBboDVhW27KZg39FSYMffVrIwrIa0gEQAbK1mdlE+iI+pLA0aQzy5rO mv9k8rY8ywjY7P79TCHNXudzSlgjvqAJShI3EdgXF0MPPPSDDEbxnHaKYuKcllpcjvs+ sIgHp2rgADe7Wzx4BUmcd2jeoPilhlIgjkd/wK8Chl4RkWjIL/v60ZLDr6q5YnXhG5kf DJpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aCzMOUKu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id re15si4604567ejb.120.2021.05.28.02.26.36; Fri, 28 May 2021 02:26:37 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=aCzMOUKu; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 26669689FFE; Fri, 28 May 2021 12:26:08 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AA453689F2C for ; Fri, 28 May 2021 12:26:04 +0300 (EEST) Received: by mail-pj1-f47.google.com with SMTP id m8-20020a17090a4148b029015fc5d36343so2190083pjg.1 for ; Fri, 28 May 2021 02:26:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r5fqZtrtZk3YHOPb3Btd4nrDLrACTsjZm0TvVHkIYIw=; b=aCzMOUKuV5jgN36LoE43cHMGfSPfYc8JfoeI8J+6OmWkFSeZwAksQe04+spMoMJJHE BxplRGnx2RzUnIYizxWgKfOv92Ef9hdr2ivbumoevbDkXLRpbx42FHxON7C0/VGS6VHZ vzRNpe1hW/aLYhCqFfJcl7ym9ytEAOK2rETAG1rsgnmpVLGmgjMzIZj8mFFtkiEiL2MC l+U95RautcqstlMmNGruwjN3l3nG1D0nyPW+49BXv3j3rZAurDoFmIS2rQJ9RFoso4W5 K65eRjGu0zx7Xyxk7M/x0N6E/b+q05UT2jz30/JtCE0BFXmRFFoPRDxW+OzT9G38Weps aNLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r5fqZtrtZk3YHOPb3Btd4nrDLrACTsjZm0TvVHkIYIw=; b=GE8A2JzHP5+k7PgfvKypyY6iQDijlFpp8cFx3gcwIziSXM59sz9MoVQhVnvfbvwyni G3oHl4cGjq/12FPMQPcTfIUsNzr1LGIlH5qF1WxNcuyZzgLDiaNISqc0o6GEcunSEJis 0F9rTeRpVSF0yZ2dH+9lzMYkGLDGjXv3hzSGGijhTMSW+SZCnVrOazKau3bTXD0es2UG hTCYCxtJAi/HmJqKoCT3R5Tp9ZI8vqJipRZtp6qB0DI2AM9UBxL6AVPfe6LGbe5L8z8b lzP5Y2qj6KUqjNbxzBfXfxwXaI+I7UGA6A59u/XycObbcXwFi+QcjG5zucRo6wtbdD3v nudA== X-Gm-Message-State: AOAM531gZB13dYH9Zpx7K7bOLIzuu8o7J2MkaR/+KfAAARxEoXsRLXIU 2NvbLiaxx63DSSWaq2ZJct0Tp/5VROSQLA== X-Received: by 2002:a17:902:728f:b029:fe:11e1:cb67 with SMTP id d15-20020a170902728fb02900fe11e1cb67mr7088117pll.23.1622193962834; Fri, 28 May 2021 02:26:02 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.26.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:26:02 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:53 +0530 Message-Id: <20210528092454.31874-9-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 09/10] lavfi/dnn: Async Support for TensorFlow Backend X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 3mYFarKvj2+n This commit adds functions to execute the inference requests to TensorFlow Backend asynchronously in detached threads. Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 198 ++++++++++++++++++++++++++++--- libavfilter/dnn/dnn_backend_tf.h | 3 + libavfilter/dnn/dnn_interface.c | 3 + 3 files changed, 187 insertions(+), 17 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 31746deef4..296604461b 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -35,6 +35,7 @@ #include "dnn_backend_native_layer_maximum.h" #include "dnn_io_proc.h" #include "dnn_backend_common.h" +#include "libavutil/thread.h" #include "safe_queue.h" #include "queue.h" #include @@ -57,6 +58,7 @@ typedef struct TFModel{ TF_Status *status; SafeQueue *request_queue; Queue *inference_queue; + Queue *task_queue; } TFModel; typedef struct tf_infer_request { @@ -69,7 +71,10 @@ typedef struct tf_infer_request { typedef struct RequestItem { tf_infer_request *infer_request; InferenceItem *inference; - // further properties will be added later for async +#if HAVE_PTHREAD_CANCEL + pthread_t thread; + pthread_attr_t thread_attr; +#endif } RequestItem; #define OFFSET(x) offsetof(TFContext, x) @@ -83,6 +88,7 @@ static const AVOption dnn_tensorflow_options[] = { AVFILTER_DEFINE_CLASS(dnn_tensorflow); static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue); +static void infer_completion_callback(void *args); static void free_buffer(void *data, size_t length) { @@ -112,6 +118,59 @@ static tf_infer_request* tf_create_inference_request(void) return infer_request; } +static void tf_start_inference(RequestItem *request) +{ + tf_infer_request *infer_request = request->infer_request; + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + TFModel *tf_model = task->model; + + TF_SessionRun(tf_model->session, NULL, + infer_request->tf_input, &infer_request->input_tensor, 1, + infer_request->tf_outputs, infer_request->output_tensors, + task->nb_output, NULL, 0, NULL, + tf_model->status); +} + +static void *tf_thread_routine(void *arg) +{ + RequestItem *request = arg; + tf_start_inference(request); + infer_completion_callback(request); +#if HAVE_PTHREAD_CANCEL + pthread_exit(0); +#endif +} + +static DNNReturnType tf_start_inference_async(RequestItem *request) +{ + InferenceItem *inference = request->inference; + TaskItem *task = inference->task; + TFModel *tf_model = task->model; + TFContext *ctx = &tf_model->ctx; + int ret; + +#if HAVE_PTHREAD_CANCEL + ret = pthread_create(&request->thread, &request->thread_attr, tf_thread_routine, request); + if (ret != 0) + { + av_log(ctx, AV_LOG_ERROR, "unable to start async inference\n"); + return DNN_ERROR; + } + return DNN_SUCCESS; +#else + av_log(ctx, AV_LOG_WARNING, "pthreads not supported. Roll back to sync\n"); + tf_start_inference(request); + if (TF_GetCode(tf_model->status) != TF_OK) { + tf_free_request(request->infer_request); + av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); + return DNN_ERROR; + } + infer_completion_callback(request); + return (task->inference_done == task->inference_todo) ? DNN_SUCCESS : DNN_ERROR; +#endif +} + static DNNReturnType extract_inference_from_task(TaskItem *task, Queue *inference_queue) { TFModel *tf_model = task->model; @@ -826,7 +885,10 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ av_freep(&item); goto err; } - +#if HAVE_PTHREAD_CANCEL + pthread_attr_init(&item->thread_attr); + pthread_attr_setdetachstate(&item->thread_attr, PTHREAD_CREATE_DETACHED); +#endif if (ff_safe_queue_push_back(tf_model->request_queue, item) < 0) { av_freep(&item->infer_request); av_freep(&item); @@ -839,6 +901,16 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_ goto err; } + tf_model->task_queue = ff_queue_create(); + if (!tf_model->task_queue) { + goto err; + } + + tf_model->inference_queue = ff_queue_create(); + if (!tf_model->inference_queue) { + goto err; + } + model->model = tf_model; model->get_input = &get_input_tf; model->get_output = &get_output_tf; @@ -1012,10 +1084,9 @@ final: static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_queue) { TFModel *tf_model; - TFContext *ctx; - tf_infer_request *infer_request; InferenceItem *inference; TaskItem *task; + TFContext *ctx; inference = ff_queue_peek_front(inference_queue); if (!inference) { @@ -1026,22 +1097,16 @@ static DNNReturnType execute_model_tf(RequestItem *request, Queue *inference_que tf_model = task->model; ctx = &tf_model->ctx; - if (task->async) { - avpriv_report_missing_feature(ctx, "Async execution not supported"); + if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { return DNN_ERROR; - } else { - if (fill_model_input_tf(tf_model, request) != DNN_SUCCESS) { - return DNN_ERROR; - } + } - infer_request = request->infer_request; - TF_SessionRun(tf_model->session, NULL, - infer_request->tf_input, &infer_request->input_tensor, 1, - infer_request->tf_outputs, infer_request->output_tensors, - task->nb_output, NULL, 0, NULL, - tf_model->status); + if (task->async) { + return tf_start_inference_async(request); + } else { + tf_start_inference(request); if (TF_GetCode(tf_model->status) != TF_OK) { - tf_free_request(infer_request); + tf_free_request(request->infer_request); av_log(ctx, AV_LOG_ERROR, "Failed to run session when executing model\n"); return DNN_ERROR; } @@ -1079,6 +1144,94 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams * return execute_model_tf(request, tf_model->inference_queue); } +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params) { + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + TaskItem *task; + RequestItem *request; + + if (ff_check_exec_params(ctx, DNN_TF, model->func_type, exec_params) != 0) { + return DNN_ERROR; + } + + task = av_malloc(sizeof(*task)); + if (!task) { + av_log(ctx, AV_LOG_ERROR, "unable to alloc memory for task item.\n"); + return DNN_ERROR; + } + + if (ff_dnn_fill_task(task, exec_params, tf_model, 1, 1) != DNN_SUCCESS) { + av_freep(&task); + return DNN_ERROR; + } + + if (ff_queue_push_back(tf_model->task_queue, task) < 0) { + av_freep(&task); + av_log(ctx, AV_LOG_ERROR, "unable to push back task_queue.\n"); + return DNN_ERROR; + } + + if (extract_inference_from_task(task, tf_model->inference_queue) != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "unable to extract inference from task.\n"); + return DNN_ERROR; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + return execute_model_tf(request, tf_model->inference_queue); +} + +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) +{ + TFModel *tf_model = model->model; + TaskItem *task = ff_queue_peek_front(tf_model->task_queue); + + if (!task) { + return DAST_EMPTY_QUEUE; + } + + if (task->inference_done != task->inference_todo) { + return DAST_NOT_READY; + } + + *in = task->in_frame; + *out = task->out_frame; + ff_queue_pop_front(tf_model->task_queue); + av_freep(&task); + + return DAST_SUCCESS; +} + +DNNReturnType ff_dnn_flush_tf(const DNNModel *model) +{ + TFModel *tf_model = model->model; + TFContext *ctx = &tf_model->ctx; + RequestItem *request; + DNNReturnType ret; + + if (ff_queue_size(tf_model->inference_queue) == 0) { + // no pending task need to flush + return DNN_SUCCESS; + } + + request = ff_safe_queue_pop_front(tf_model->request_queue); + if (!request) { + av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n"); + return DNN_ERROR; + } + + ret = fill_model_input_tf(tf_model, request); + if (ret != DNN_SUCCESS) { + av_log(ctx, AV_LOG_ERROR, "Failed to fill model input.\n"); + return ret; + } + + return tf_start_inference_async(request); +} + void ff_dnn_free_model_tf(DNNModel **model) { TFModel *tf_model; @@ -1087,6 +1240,9 @@ void ff_dnn_free_model_tf(DNNModel **model) tf_model = (*model)->model; while (ff_safe_queue_size(tf_model->request_queue) != 0) { RequestItem *item = ff_safe_queue_pop_front(tf_model->request_queue); +#if HAVE_PTHREAD_CANCEL + pthread_attr_destroy(&item->thread_attr); +#endif tf_free_request(item->infer_request); av_freep(&item->infer_request); av_freep(&item); @@ -1099,6 +1255,14 @@ void ff_dnn_free_model_tf(DNNModel **model) } ff_queue_destroy(tf_model->inference_queue); + while (ff_queue_size(tf_model->task_queue) != 0) { + TaskItem *item = ff_queue_pop_front(tf_model->task_queue); + av_frame_free(&item->in_frame); + av_frame_free(&item->out_frame); + av_freep(&item); + } + ff_queue_destroy(tf_model->task_queue); + if (tf_model->graph){ TF_DeleteGraph(tf_model->graph); } diff --git a/libavfilter/dnn/dnn_backend_tf.h b/libavfilter/dnn/dnn_backend_tf.h index 3dfd6e4280..aec0fc2011 100644 --- a/libavfilter/dnn/dnn_backend_tf.h +++ b/libavfilter/dnn/dnn_backend_tf.h @@ -32,6 +32,9 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename, DNNFunctionType func_type, const char *options, AVFilterContext *filter_ctx); DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNReturnType ff_dnn_execute_model_async_tf(const DNNModel *model, DNNExecBaseParams *exec_params); +DNNAsyncStatusType ff_dnn_get_async_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out); +DNNReturnType ff_dnn_flush_tf(const DNNModel *model); void ff_dnn_free_model_tf(DNNModel **model); diff --git a/libavfilter/dnn/dnn_interface.c b/libavfilter/dnn/dnn_interface.c index 02e532fc1b..81af934dd5 100644 --- a/libavfilter/dnn/dnn_interface.c +++ b/libavfilter/dnn/dnn_interface.c @@ -48,6 +48,9 @@ DNNModule *ff_get_dnn_module(DNNBackendType backend_type) #if (CONFIG_LIBTENSORFLOW == 1) dnn_module->load_model = &ff_dnn_load_model_tf; dnn_module->execute_model = &ff_dnn_execute_model_tf; + dnn_module->execute_model_async = &ff_dnn_execute_model_async_tf; + dnn_module->get_async_result = &ff_dnn_get_async_result_tf; + dnn_module->flush = &ff_dnn_flush_tf; dnn_module->free_model = &ff_dnn_free_model_tf; #else av_freep(&dnn_module); From patchwork Fri May 28 09:24:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubhanshu Saxena X-Patchwork-Id: 27963 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:b214:0:0:0:0:0 with SMTP id b20csp368767iof; Fri, 28 May 2021 02:26:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxsviZln4iRQcfjXdANFbLU8CWhwuLDZBFkJkk1oWae0/dxnocJxbJcsDe92uLQf37fs5wQ X-Received: by 2002:a17:906:7696:: with SMTP id o22mr7813555ejm.298.1622194019479; Fri, 28 May 2021 02:26:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1622194019; cv=none; d=google.com; s=arc-20160816; b=uTbZMVCQbzTDYlruBYhLkoPlEjF0cGe6SMbNvXZeTzTF2pXD3rDhdDbxJ1DVMC8J+e shIhB9cY+38UHBu7mi/fT35c4aRxduZDD3dVum10sb8fdNFZ1pTZ/O3b40CXY5sYFFqN dryHFEksh5zCPSCkCqpCs8XRtfgOaA0jJQBlfTXY8uP8c4Aos8ITl5yF+ewh4D6WN55E q36+HDt8v3ZL0GVeQP9KKKWcEKT/hoo+nkz+JYZwpQlI7TznXPgAdHiHoZoMIJZuUrvQ aDT1JuDOUBeG4sVT27EH2xoIqYrXj1P5usfpJvqzQMMjUjQ0eaHgHIhdD7gnTfknWEzZ ZStw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=QGTPrxVKJS7d9CAOXJFpkKQ5oeM+//NykkEqZDjzdhI=; b=txXQYJt8iLHDiwQbGoT6h9PZCj9tfjbrSmLHWj4QyM5sWJQsbQeMKrj2mhUf3oBezB wfRTowAMxoI9HtWb1Bzg0BJwX4yEstgwV+qY43fj4MeqM+weMNfaKJN2lqlwZdAgLoyu v8Ov6BjM0JzLgQYjxWvMEJj92fX3Ts9ZVnFt9vDdjlS4iYjjWhXfA+4AYbBdaYM3KMRu bPontiHSK13H48XkhX3pLAjOSW72NNdDqzk3PtSInlbgFZenmxbSaqcACPg496gNUSHE f9fOVkXOQu0hELC9PRndShT+IIUoozytNtQIx4R2I0oexVhpGBB8GKcTMcj6AJUNtMSx a/GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=eO6lUMyx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dv20si4434113ejb.404.2021.05.28.02.26.59; Fri, 28 May 2021 02:26:59 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=eO6lUMyx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC11668A0C5; Fri, 28 May 2021 12:26:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 64C96680B73 for ; Fri, 28 May 2021 12:26:06 +0300 (EEST) Received: by mail-pj1-f49.google.com with SMTP id f3-20020a17090a4a83b02901619627235bso3485pjh.1 for ; Fri, 28 May 2021 02:26:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ELBHTE0auQak5nvJfoL4lGznJ8OzJrQ/EFWfA4V0nA=; b=eO6lUMyxRGnra0ksCdZ/FvKtCC0hXlOiGFzbGGOs/ohPwMd0alhZz8aOfQNKUmbSj6 F/Bihcg94o+qtguPDINs8e+haFntQVEH69hkGNSxtKLvfNLuREzelcapltlDtF0a4bOW tjBR6eZ7sLVDaQ4AgFNbLAyAKauRL68i01QWH7d4DQQG7pllDYWHmeWOmKy3OwLdp49m Ta6YFS/fQs/KR3Gm6cGbYD080j3VIFupQUoyeSacbno3qUbm/q3ZlFkgoNVU4VZZ3IDH NuLGxnKAZs7ElY7xyqoFdzSvZESSwW6woZus0XwVnmhNWadcEX1cB4hWbAOpt4QKZYwh CFCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ELBHTE0auQak5nvJfoL4lGznJ8OzJrQ/EFWfA4V0nA=; b=a+k7b1RLGghBTuuEFCVUmhh5pe/o6gS9NIIRwD07HuMQ03lvI53x3qZt4rwgrvCS4+ 1JHLhUpuYZRNW0YQf3OhaExbmfhipm9P2EDDKJwrnTZbI4Er4E65PJMKX3GuKea8N4aV ZPhEnTA2rpLKA90fo9Z9EofRTMLogmqT6D6SNnoLt04NLRmVy1i6BzUPlqWvSKDJzON4 bV3+MIWFMGAE4PtpR7vE/MTvm+l9zufB/gM44Pab22cEh9Lq3cEf11ca9qIMsC1SR5IG W7zvqFQAlQC+Hg35rrvMdGDueJZwIAmVx8RYI1kB7jYQY1cbrwG+KYCp9xjVlXE0K7gg Bo1g== X-Gm-Message-State: AOAM5334m7F+utdgMNkCYqwv+DujAGNlPJfxQJKcDaA8daNP3S7IGJD8 4kOCeVVMtHtd1CuXTMDn3IS+qqtvSH0LwQ== X-Received: by 2002:a17:90a:a386:: with SMTP id x6mr3555250pjp.193.1622193964647; Fri, 28 May 2021 02:26:04 -0700 (PDT) Received: from Pavilion-x360.bbrouter ([103.133.121.241]) by smtp.googlemail.com with ESMTPSA id q24sm3846892pgk.32.2021.05.28.02.26.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 02:26:04 -0700 (PDT) From: Shubhanshu Saxena To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 May 2021 14:54:54 +0530 Message-Id: <20210528092454.31874-10-shubhanshu.e01@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210528092454.31874-1-shubhanshu.e01@gmail.com> References: <20210528092454.31874-1-shubhanshu.e01@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 10/10] lavfi/dnn_backend_tf.c: Documentation for tf_infer_request functions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Shubhanshu Saxena Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: uYl3fav9cUGY Documentation for functions related to tf_infer_request Signed-off-by: Shubhanshu Saxena --- libavfilter/dnn/dnn_backend_tf.c | 41 ++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index 296604461b..8a74b11cf5 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -95,6 +95,13 @@ static void free_buffer(void *data, size_t length) av_freep(&data); } +/** + * Free the contents of TensorFlow inference request. + * It does not free the tf_infer_request instance. + * + * @param request pointer to tf_infer_request instance. + * NULL pointer is allowed. + */ static void tf_free_request(tf_infer_request *request) { if (!request) @@ -108,6 +115,12 @@ static void tf_free_request(tf_infer_request *request) av_freep(&request->output_tensors); } +/** + * Create a TensorFlow inference request. All properties + * are initially unallocated and set as NULL. + * + * @return pointer to the allocated tf_infer_request instance. + */ static tf_infer_request* tf_create_inference_request(void) { tf_infer_request* infer_request = av_malloc(sizeof(tf_infer_request)); @@ -118,8 +131,17 @@ static tf_infer_request* tf_create_inference_request(void) return infer_request; } +/** + * Start synchronous inference for the TensorFlow model. + * It does not check for the status of the operation. + * Check using tf_model->status. + * + * @param request pointer to the RequestItem for inference + */ static void tf_start_inference(RequestItem *request) { + if (!request) + return; tf_infer_request *infer_request = request->infer_request; InferenceItem *inference = request->inference; TaskItem *task = inference->task; @@ -132,6 +154,12 @@ static void tf_start_inference(RequestItem *request) tf_model->status); } +/** + * Thread routine for async inference. It calls completion + * callback on completion of inference. + * + * @param arg pointer to RequestItem instance for inference + */ static void *tf_thread_routine(void *arg) { RequestItem *request = arg; @@ -142,8 +170,21 @@ static void *tf_thread_routine(void *arg) #endif } +/** + * Start asynchronous inference routine for the TensorFlow + * model on a detached thread. It calls the completion callback + * after the inference completes. + * In case pthreads aren't supported, the execution rolls back + * to synchronous mode, calling completion callback after inference. + * + * @param request pointer to the RequestItem for inference + * @retval DNN_SUCCESS on the start of async inference. + * @retval DNN_ERROR in case async inference cannot be started + */ static DNNReturnType tf_start_inference_async(RequestItem *request) { + if (!request) + return DNN_ERROR; InferenceItem *inference = request->inference; TaskItem *task = inference->task; TFModel *tf_model = task->model;