From patchwork Tue Jun 5 23:23:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Lavrushkin X-Patchwork-Id: 9264 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:11c:0:0:0:0:0 with SMTP id c28-v6csp119398jad; Tue, 5 Jun 2018 16:29:18 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK9OntxLVAaLDcMxlhvJkXdSUZlnutx2z4kBCKqfL+ZuxVx8t1Faix2P/U/Mt0sPWqa5iOA X-Received: by 2002:adf:9b86:: with SMTP id d6-v6mr460131wrc.240.1528241358434; Tue, 05 Jun 2018 16:29:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528241358; cv=none; d=google.com; s=arc-20160816; b=Qb6apdojHMey5Z7sppale/3flI3hZSpD1ph7SnnRU8LtWzigmt0BswWU4+i8prDFK/ GZDT/J5lvom+LSsnFUBsWxLz+BfN1q93R1oU0fAJ6SXdFqVIIZW23lQZ7eFcDFN0SbF7 wqLOrwLjs7nVTWva+m3nTmSCPyGBUhuOEUJwg/FX556CFmiQj4T40cOn2/a88J8C+WkJ JQR2wYl78/ROOXkR8IfpwiFQAEkYnFgd9WPq4HGlrU/76Ce5+e13JQ9FosVmWlK6a/XR 0pRv4KMvQ27593l17nOB+uQj3w5PpVyDqVqvEO4ullK1v0d8vs8dVmGx7IV2k5EQbitb VGbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence:subject:to :message-id:date:from:references:in-reply-to:mime-version :dkim-signature:delivered-to:arc-authentication-results; bh=yiyX/MeGK/EtVsRmbDDZe/IxiEFEnvWleqH+eYHuBgM=; b=dyic0YK041iXT5gkf7lAUGb1o75FueQw2p56RHAiz+uK1zJ7D50gGTe2Dfl+GDPH0F b5XGbNy8INZcN5qv2koMVFXU1/vX4dTmlOxHLkdB7D3fgPb38ra7+VP/nlUN2fu82gx7 MlGaOC1cwt3OibNRkV4X2703ro+U/eNv/AG6Y1KQrppFw0Azhl06gCS1I+3yeuNQ9ojt IsbUIx5vCVvgO0UUllI/hh03okg+Z89S791FkgPmGMjoI5x1N34pDmapIvgvutvpWmVQ vLm4oJGIhMtMYhyCTS9ROSPa91/mYTQJ0qWNCt1uC1bcxBEHe5k4Fjx+GLo9EfMTK28m NneQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OB9whRPx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m14-v6si27243778wrf.250.2018.06.05.16.29.17; Tue, 05 Jun 2018 16:29:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=OB9whRPx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 60D1F68A541; Wed, 6 Jun 2018 02:28:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf0-f196.google.com (mail-pf0-f196.google.com [209.85.192.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 3C21868A489 for ; Wed, 6 Jun 2018 02:28:21 +0300 (EEST) Received: by mail-pf0-f196.google.com with SMTP id w7-v6so2147238pfn.9 for ; Tue, 05 Jun 2018 16:29:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=V6Slns/6E+WDQlXl8aknZLzijvv8l79xli6AWX4++G0=; b=OB9whRPxnoH3TMlqXCcJDxGSCKcBXYIG9INaSveUIewgoi8Fu7Q9r1Dc/g704Bz3fP RFXN57/VTsG9TMB3kOTrpsoaPUPZEGq4j8+j8MwM79chGG9SczzN7WVeFJ8uXK0jfnoE 4i1Rlswb54hulj5QIGk/gEspNgm3HTMgpnxRgTs5zYd5AeqBUql45C93dvAZ0POYJHOu +g6AhrB/DKrWgYoYqYcTJVyZcUwa8j2yh54a8Ju3mPUCq9uYHEsGzAsqo+OuP0lK5jkg O2wTEOPwafHVujyUwysE4wMmX8lHWNjSEeqyeedGnwnmTYmRrVJTrFNp2uf8ZszFTToE Cxdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=V6Slns/6E+WDQlXl8aknZLzijvv8l79xli6AWX4++G0=; b=nmZumhcEEdtdVO3kCQzi+XRpJN1egZ0za+LzbAX3dw4Gv33bxGZQXHGQP2yceWdwWK KCtNTzW/FvdgL4OQNGNFINSJ4lkMjGRJc19D4ZCJdGyJomx+CqCI0ksfKbKNIO5gM3qY wMVQwxrXCdFEEe4HyDoGijLzpp+pjxmyMPmGA6kFqAm1zhP9xNhXUF7XNzHaEh42d92x IfsXrE3E5451LMpc4b3ltvylRkKy3vZqN6LkaCEuXpAOreg3B4jyvhNk/rvdpZZ59peX 2fZ4NIC2AwiVOIscW/7opbXtZeThgAp/nOuvHbjKIY+TAkDoysw1uWUXJ5DkvDxuQaaq z0Dw== X-Gm-Message-State: APt69E2SElp7zgmXnhkeaiQugEZYvZiCeGme1gieb3YoiA1STTMW/yVu XgFF6ls7fw7n8TKzgdTNi4gH2yRMlmUCorx9i63DFQ== X-Received: by 2002:a62:484d:: with SMTP id v74-v6mr602148pfa.256.1528241004299; Tue, 05 Jun 2018 16:23:24 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a17:90a:a607:0:0:0:0 with HTTP; Tue, 5 Jun 2018 16:23:23 -0700 (PDT) In-Reply-To: <256a62b9-edcc-83cf-6f6d-18308ee68beb@gmail.com> References: <3cf9936c-9995-1ea5-3bab-661eb644becd@gmail.com> <256a62b9-edcc-83cf-6f6d-18308ee68beb@gmail.com> From: Sergey Lavrushkin Date: Wed, 6 Jun 2018 02:23:23 +0300 Message-ID: To: FFmpeg development discussions and patches X-Content-Filtered-By: Mailman/MimeDel 2.1.20 Subject: Re: [FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: James Almer Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" 2018-06-05 17:20 GMT+03:00 James Almer : > On 6/3/2018 3:02 PM, Sergey Lavrushkin wrote: > > diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c > > index d6efe9b478..5c5e26b33a 100644 > > --- a/libavfilter/vf_srcnn.c > > +++ b/libavfilter/vf_srcnn.c > > @@ -41,7 +41,6 @@ typedef struct SRCNNContext { > > DNNData input_output; > > } SRCNNContext; > > > > - > > #define OFFSET(x) offsetof(SRCNNContext, x) > > #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM > > static const AVOption srcnn_options[] = { > > @@ -55,10 +54,19 @@ static av_cold int init(AVFilterContext* context) > > { > > SRCNNContext* srcnn_context = context->priv; > > > > - srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE); > > + srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF); > > This should be a filter AVOption, not hardcoded to one or another. What > if i, for whatever reason, want to use the native backend when i have > libtensorflow enabled? > > > if (!srcnn_context->dnn_module){ > > - av_log(context, AV_LOG_ERROR, "could not create dnn module\n"); > > - return AVERROR(ENOMEM); > > + srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE); > > + if (!srcnn_context->dnn_module){ > > + av_log(context, AV_LOG_ERROR, "could not create dnn > module\n"); > > + return AVERROR(ENOMEM); > > + } > > + else{ > > + av_log(context, AV_LOG_INFO, "using native backend for DNN > inference\n"); > > VERBOSE, not INFO > > > + } > > + } > > + else{ > > + av_log(context, AV_LOG_INFO, "using tensorflow backend for DNN > inference\n"); > > Ditto. > > > } > > if (!srcnn_context->model_filename){ > > av_log(context, AV_LOG_INFO, "model file for network was not > specified, using default network for x2 upsampling\n"); Here is the patch, that fixes described issues. From 971e15b4b1e3f2747aa07d0221f99226cba622ac Mon Sep 17 00:00:00 2001 From: Sergey Lavrushkin Date: Wed, 6 Jun 2018 01:44:40 +0300 Subject: [PATCH] libavfilter/vf_srcnn.c: adds DNN module backend AVOption, changes AV_LOG_INFO message to AV_LOG_VERBOSE. --- libavfilter/vf_srcnn.c | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/libavfilter/vf_srcnn.c b/libavfilter/vf_srcnn.c index 5c5e26b33a..17e380503e 100644 --- a/libavfilter/vf_srcnn.c +++ b/libavfilter/vf_srcnn.c @@ -36,6 +36,7 @@ typedef struct SRCNNContext { char* model_filename; float* input_output_buf; + DNNBackendType backend_type; DNNModule* dnn_module; DNNModel* model; DNNData input_output; @@ -44,6 +45,9 @@ typedef struct SRCNNContext { #define OFFSET(x) offsetof(SRCNNContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM static const AVOption srcnn_options[] = { + { "dnn_backend", "DNN backend used for model execution", OFFSET(backend_type), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, 1, FLAGS, "backend" }, + { "native", "native backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, 0, 0, FLAGS, "backend" }, + { "tensorflow", "tensorflow backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, 0, 0, FLAGS, "backend" }, { "model_filename", "path to model file specifying network architecture and its parameters", OFFSET(model_filename), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, { NULL } }; @@ -54,29 +58,20 @@ static av_cold int init(AVFilterContext* context) { SRCNNContext* srcnn_context = context->priv; - srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF); + srcnn_context->dnn_module = ff_get_dnn_module(srcnn_context->backend_type); if (!srcnn_context->dnn_module){ - srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE); - if (!srcnn_context->dnn_module){ - av_log(context, AV_LOG_ERROR, "could not create dnn module\n"); - return AVERROR(ENOMEM); - } - else{ - av_log(context, AV_LOG_INFO, "using native backend for DNN inference\n"); - } - } - else{ - av_log(context, AV_LOG_INFO, "using tensorflow backend for DNN inference\n"); + av_log(context, AV_LOG_ERROR, "could not create DNN module for requested backend\n"); + return AVERROR(ENOMEM); } if (!srcnn_context->model_filename){ - av_log(context, AV_LOG_INFO, "model file for network was not specified, using default network for x2 upsampling\n"); + av_log(context, AV_LOG_VERBOSE, "model file for network was not specified, using default network for x2 upsampling\n"); srcnn_context->model = (srcnn_context->dnn_module->load_default_model)(DNN_SRCNN); } else{ srcnn_context->model = (srcnn_context->dnn_module->load_model)(srcnn_context->model_filename); } if (!srcnn_context->model){ - av_log(context, AV_LOG_ERROR, "could not load dnn model\n"); + av_log(context, AV_LOG_ERROR, "could not load DNN model\n"); return AVERROR(EIO); } -- 2.14.1