From patchwork Thu Aug 2 18:52:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Lavrushkin X-Patchwork-Id: 9871 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp2401712jad; Thu, 2 Aug 2018 12:00:57 -0700 (PDT) X-Google-Smtp-Source: AAOMgpd9V/tjZ2KQ3pn1X0f9crEvJYTEEmbZ7kNXFZh/YbkE4WvAklKZFoZAZijmEr5yoVBCkEGS X-Received: by 2002:a7b:c041:: with SMTP id u1-v6mr2903878wmc.53.1533236457554; Thu, 02 Aug 2018 12:00:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533236457; cv=none; d=google.com; s=arc-20160816; b=GmcoNR3zKoz8paJQtyIwvHwRES2xroFHNo+W2K65uL4oi/JNmB4Td00myZF9FCa/gw kLmm8spveVJaPqp8/akeNFvLx/is6DsCH+uIaOPsLGxY75dgbynyqbOhge2wUHv4QBuy t7OdnVHHZZIgeqbR63Htzc8YZygEaFRjC84m2sXxQ38NF8GKXnMR9jpoYcbWtvS1aZCq yPH8z8W/Q53O/7YPjMcbMIWjUxgFAvwwMhD3VGFDzp1UomGxTLZesD+BNzB9S4PTWYsE E3drn+SegsdEnGgKyQrzsaG5sASH3UisEVbJgMwZMj/zloQ+mnyoQ6AZCqY85dKlPz6M uMFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=2R8F7Yz9LDBBwEe3WdTmia8UM6e13TZvdQpWP7UQTaU=; b=rqyCmld5W/IGDR+zfFgL40Esl/Tk1M2+M3RwjZF7kIEerzpngIKzAsSJEEJkj3yHGw Cns/FuYvXI3KjRbxi1Cg79U8lvhvMAlCx43oNV7idakh7aO9HNKUPmdlkPfn2RiBNANI w9iHf/6bWsssWJNaDWF3JVjdG0Oqh2wPGFb4y7TO65Zc7QzKh2Lzacna4OZwlbp4MXk5 x52hRxlGtZQfRBdYQtYbsm8I07YQzLS1SPC2KAPj3/zbfq+Yz3KRyVYadK92SFM+4ZNM 6tEVE4l+rGXd4zgivaGYPV0XMQ3bH+nzIKJJfkf8S4WYtNY3tSWZbbN2WM/Pd0eaOm1N FgVw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=L89zzPNP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a185-v6si1971388wmd.193.2018.08.02.12.00.56; Thu, 02 Aug 2018 12:00:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=L89zzPNP; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6C29368A40C; Thu, 2 Aug 2018 22:00:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 187A1689AF7 for ; Thu, 2 Aug 2018 22:00:31 +0300 (EEST) Received: by mail-lj1-f178.google.com with SMTP id w16-v6so2809783ljh.12 for ; Thu, 02 Aug 2018 12:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=oDQssgnitMNiD4RxA6trg3l8WHEGey1WMWmIJiVqdR8=; b=L89zzPNPyoyhgh5xUoNLDx96wkNQLkxXAl7DzJzT/fW+nmw5oquL8mGNj24M9/TmWh n0NF/g5MQdnN/N6u0Ryp6E48IhL0J9neA6ottitD3LMqvo9+Q9TA8WJ1vCa6HTfikoby zy769saewTcG0QRUW46BsSOVcw1PRN18yF+CVPYMtUYd9NNAu/HOdDejMz2EELych+Im bBgnXrkAESk12DwcvsTvF5Rr07+YvEZWFOuJyKsljXiVZMn+v3pKyTRUY6SNDev/9pFM 4ehw1yAu4aZZqge1I1WCpMClHjHEVZyHCpTwNxXpIJMThT1WMUlVll+mPyM6mz4AYTdN 5veg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=oDQssgnitMNiD4RxA6trg3l8WHEGey1WMWmIJiVqdR8=; b=hu41R/kUKLoc8ZRtnU5O4wXgAtXcKXpj4xVKLsYycf/ImrzxeWzp3DsvxrSStstYqT WLm2ZSbL7LUcb5Cnk+95di7lOxPNQ4bI1n0DNa2nBwGVBJIHBNXmOFoJQG76d0AgtzSi XXEfkSjE62Gbh9MQ6ijz9zGJaMusYcqliuVQC3siqlrgFuQUpQIjFB41gRgFNGDdjdmE NXe7pWBsJCNz6tjkFnI12u3OtoU24p0yM0u61YEsn5xBiW+uKVP02XYOQ6xlLMwXMJgD CtmJSx8IsFtmAIKNRTCtsLGIL6l0UfFqpBHC7tgZTmW/QqPZDPom1adRGPt9dcu16R8W DE2Q== X-Gm-Message-State: AOUpUlFeVhicWOc58dLaA4n5LBsrYRpK92yNXgOyfPzTjXMBExtF1jt6 rjTA6Ur3g3D3IL6H8KOiZKJipmy2 X-Received: by 2002:a2e:8e8c:: with SMTP id z12-v6mr2994538ljk.117.1533236066746; Thu, 02 Aug 2018 11:54:26 -0700 (PDT) Received: from localhost.localdomain (nat-1-27.msu.umos.ru. [85.89.127.27]) by smtp.gmail.com with ESMTPSA id h4-v6sm410377lfj.69.2018.08.02.11.54.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Aug 2018 11:54:25 -0700 (PDT) From: Sergey Lavrushkin To: ffmpeg-devel@ffmpeg.org Date: Thu, 2 Aug 2018 21:52:47 +0300 Message-Id: <20180802185248.18168-7-dualfal@gmail.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180802185248.18168-1-dualfal@gmail.com> References: <20180802185248.18168-1-dualfal@gmail.com> Subject: [FFmpeg-devel] [PATCH 6/7] libavfilter/vf_sr.c: Removes uint8 -> float and float -> uint8 conversions. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Sergey Lavrushkin MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This patch removes conversions, declared inside the sr filter, and uses libswscale inside the filter to perform them for only Y channel of input. The sr filter still has uint formats as input, as it does not use chroma channels in models and these channels are upscaled using libswscale, float formats for input would cause unnecessary conversions during scaling for these channels. --- libavfilter/vf_sr.c | 134 +++++++++++++++++++--------------------------------- 1 file changed, 48 insertions(+), 86 deletions(-) diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 944a0e28e7..5ad1baa4c0 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -45,8 +45,8 @@ typedef struct SRContext { DNNModel *model; DNNData input, output; int scale_factor; - struct SwsContext *sws_context; - int sws_slice_h; + struct SwsContext *sws_contexts[3]; + int sws_slice_h, sws_input_linesize, sws_output_linesize; } SRContext; #define OFFSET(x) offsetof(SRContext, x) @@ -95,6 +95,10 @@ static av_cold int init(AVFilterContext *context) return AVERROR(EIO); } + sr_context->sws_contexts[0] = NULL; + sr_context->sws_contexts[1] = NULL; + sr_context->sws_contexts[2] = NULL; + return 0; } @@ -110,6 +114,7 @@ static int query_formats(AVFilterContext *context) av_log(context, AV_LOG_ERROR, "could not create formats list\n"); return AVERROR(ENOMEM); } + return ff_set_common_formats(context, formats_list); } @@ -140,21 +145,31 @@ static int config_props(AVFilterLink *inlink) else{ outlink->h = sr_context->output.height; outlink->w = sr_context->output.width; + sr_context->sws_contexts[1] = sws_getContext(sr_context->input.width, sr_context->input.height, AV_PIX_FMT_GRAY8, + sr_context->input.width, sr_context->input.height, AV_PIX_FMT_GRAYF32, + 0, NULL, NULL, NULL); + sr_context->sws_input_linesize = sr_context->input.width << 2; + sr_context->sws_contexts[2] = sws_getContext(sr_context->output.width, sr_context->output.height, AV_PIX_FMT_GRAYF32, + sr_context->output.width, sr_context->output.height, AV_PIX_FMT_GRAY8, + 0, NULL, NULL, NULL); + sr_context->sws_output_linesize = sr_context->output.width << 2; + if (!sr_context->sws_contexts[1] || !sr_context->sws_contexts[2]){ + av_log(context, AV_LOG_ERROR, "could not create SwsContext for conversions\n"); + return AVERROR(ENOMEM); + } switch (sr_context->model_type){ case SRCNN: - sr_context->sws_context = sws_getContext(inlink->w, inlink->h, inlink->format, - outlink->w, outlink->h, outlink->format, SWS_BICUBIC, NULL, NULL, NULL); - if (!sr_context->sws_context){ - av_log(context, AV_LOG_ERROR, "could not create SwsContext\n"); + sr_context->sws_contexts[0] = sws_getContext(inlink->w, inlink->h, inlink->format, + outlink->w, outlink->h, outlink->format, + SWS_BICUBIC, NULL, NULL, NULL); + if (!sr_context->sws_contexts[0]){ + av_log(context, AV_LOG_ERROR, "could not create SwsContext for scaling\n"); return AVERROR(ENOMEM); } sr_context->sws_slice_h = inlink->h; break; case ESPCN: - if (inlink->format == AV_PIX_FMT_GRAY8){ - sr_context->sws_context = NULL; - } - else{ + if (inlink->format != AV_PIX_FMT_GRAY8){ sws_src_h = sr_context->input.height; sws_src_w = sr_context->input.width; sws_dst_h = sr_context->output.height; @@ -184,13 +199,14 @@ static int config_props(AVFilterLink *inlink) sws_dst_w = AV_CEIL_RSHIFT(sws_dst_w, 2); break; default: - av_log(context, AV_LOG_ERROR, "could not create SwsContext for input pixel format"); + av_log(context, AV_LOG_ERROR, "could not create SwsContext for scaling for given input pixel format"); return AVERROR(EIO); } - sr_context->sws_context = sws_getContext(sws_src_w, sws_src_h, AV_PIX_FMT_GRAY8, - sws_dst_w, sws_dst_h, AV_PIX_FMT_GRAY8, SWS_BICUBIC, NULL, NULL, NULL); - if (!sr_context->sws_context){ - av_log(context, AV_LOG_ERROR, "could not create SwsContext\n"); + sr_context->sws_contexts[0] = sws_getContext(sws_src_w, sws_src_h, AV_PIX_FMT_GRAY8, + sws_dst_w, sws_dst_h, AV_PIX_FMT_GRAY8, + SWS_BICUBIC, NULL, NULL, NULL); + if (!sr_context->sws_contexts[0]){ + av_log(context, AV_LOG_ERROR, "could not create SwsContext for scaling\n"); return AVERROR(ENOMEM); } sr_context->sws_slice_h = sws_src_h; @@ -201,61 +217,12 @@ static int config_props(AVFilterLink *inlink) } } -typedef struct ThreadData{ - uint8_t *data; - int data_linesize, height, width; -} ThreadData; - -static int uint8_to_float(AVFilterContext *context, void *arg, int jobnr, int nb_jobs) -{ - SRContext *sr_context = context->priv; - const ThreadData *td = arg; - const int slice_start = (td->height * jobnr ) / nb_jobs; - const int slice_end = (td->height * (jobnr + 1)) / nb_jobs; - const uint8_t *src = td->data + slice_start * td->data_linesize; - float *dst = sr_context->input.data + slice_start * td->width; - int y, x; - - for (y = slice_start; y < slice_end; ++y){ - for (x = 0; x < td->width; ++x){ - dst[x] = (float)src[x] / 255.0f; - } - src += td->data_linesize; - dst += td->width; - } - - return 0; -} - -static int float_to_uint8(AVFilterContext *context, void *arg, int jobnr, int nb_jobs) -{ - SRContext *sr_context = context->priv; - const ThreadData *td = arg; - const int slice_start = (td->height * jobnr ) / nb_jobs; - const int slice_end = (td->height * (jobnr + 1)) / nb_jobs; - const float *src = sr_context->output.data + slice_start * td->width; - uint8_t *dst = td->data + slice_start * td->data_linesize; - int y, x; - - for (y = slice_start; y < slice_end; ++y){ - for (x = 0; x < td->width; ++x){ - dst[x] = (uint8_t)(255.0f * FFMIN(src[x], 1.0f)); - } - src += td->width; - dst += td->data_linesize; - } - - return 0; -} - static int filter_frame(AVFilterLink *inlink, AVFrame *in) { AVFilterContext *context = inlink->dst; SRContext *sr_context = context->priv; AVFilterLink *outlink = context->outputs[0]; AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); - ThreadData td; - int nb_threads; DNNReturnType dnn_result; if (!out){ @@ -268,28 +235,23 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) out->width = sr_context->output.width; switch (sr_context->model_type){ case SRCNN: - sws_scale(sr_context->sws_context, (const uint8_t **)in->data, in->linesize, + sws_scale(sr_context->sws_contexts[0], (const uint8_t **)in->data, in->linesize, 0, sr_context->sws_slice_h, out->data, out->linesize); - td.data = out->data[0]; - td.data_linesize = out->linesize[0]; - td.height = out->height; - td.width = out->width; + + sws_scale(sr_context->sws_contexts[1], (const uint8_t **)out->data, out->linesize, + 0, out->height, (uint8_t * const*)(&sr_context->input.data), &sr_context->sws_input_linesize); break; case ESPCN: - if (sr_context->sws_context){ - sws_scale(sr_context->sws_context, (const uint8_t **)(in->data + 1), in->linesize + 1, + if (sr_context->sws_contexts[0]){ + sws_scale(sr_context->sws_contexts[0], (const uint8_t **)(in->data + 1), in->linesize + 1, 0, sr_context->sws_slice_h, out->data + 1, out->linesize + 1); - sws_scale(sr_context->sws_context, (const uint8_t **)(in->data + 2), in->linesize + 2, + sws_scale(sr_context->sws_contexts[0], (const uint8_t **)(in->data + 2), in->linesize + 2, 0, sr_context->sws_slice_h, out->data + 2, out->linesize + 2); } - td.data = in->data[0]; - td.data_linesize = in->linesize[0]; - td.height = in->height; - td.width = in->width; - } - nb_threads = ff_filter_get_nb_threads(context); - context->internal->execute(context, uint8_to_float, &td, NULL, FFMIN(td.height, nb_threads)); + sws_scale(sr_context->sws_contexts[1], (const uint8_t **)in->data, in->linesize, + 0, in->height, (uint8_t * const*)(&sr_context->input.data), &sr_context->sws_input_linesize); + } av_frame_free(&in); dnn_result = (sr_context->dnn_module->execute_model)(sr_context->model); @@ -298,17 +260,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) return AVERROR(EIO); } - td.data = out->data[0]; - td.data_linesize = out->linesize[0]; - td.height = out->height; - td.width = out->width; - context->internal->execute(context, float_to_uint8, &td, NULL, FFMIN(td.height, nb_threads)); + sws_scale(sr_context->sws_contexts[2], (const uint8_t **)(&sr_context->output.data), &sr_context->sws_output_linesize, + 0, out->height, (uint8_t * const*)out->data, out->linesize); return ff_filter_frame(outlink, out); } static av_cold void uninit(AVFilterContext *context) { + int i; SRContext *sr_context = context->priv; if (sr_context->dnn_module){ @@ -316,8 +276,10 @@ static av_cold void uninit(AVFilterContext *context) av_freep(&sr_context->dnn_module); } - if (sr_context->sws_context){ - sws_freeContext(sr_context->sws_context); + for (i = 0; i < 3; ++i){ + if (sr_context->sws_contexts[i]){ + sws_freeContext(sr_context->sws_contexts[i]); + } } }