From patchwork Thu Dec 31 15:17:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 24729 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 128B344A58A for ; Thu, 31 Dec 2020 17:44:07 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E086B680C53; Thu, 31 Dec 2020 17:44:06 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8BCDD680C53 for ; Thu, 31 Dec 2020 17:43:59 +0200 (EET) Received: by mail-wr1-f54.google.com with SMTP id m5so20328743wrx.9 for ; Thu, 31 Dec 2020 07:43:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=xK2ATzKBqdVIEVGQdUM/sE4OuNNpg7dAoOnDLWFA0x4=; b=hb6V3N2SAvBbGqlX4AuySSHS/Vu4INnYW5hi5t/rTjIt8OLsmnGvXqgH7ceQ4flf9F 3/0d+PuOuBn+0iBoNrVOocJXsG3OYq7qp8n1PjccTMcg6u8BwXQh39MOugxNmnbt7Pkh +sP6WZNWUnpK7Bm3ydHW0lIMSconJ4qaTTWybdvJSG6EK3u7L6rNGLLCskTY90HgVw59 owR6W+1KQ+6J7qQ0/c5nWAM/8y9EYBCduqK23/9RqdCr7XxRGOqomFbONr814y3/jCh4 HytArG0iVfVdp0IefRmvIrDdWUHcQOX2g1iq0GMvITZfdLco3CQMpOYS04U7JmzYBND5 8orw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=xK2ATzKBqdVIEVGQdUM/sE4OuNNpg7dAoOnDLWFA0x4=; b=Kek9kPnpdkktoHLxXtJkORacLuHCgZTDMIT1ADX6qJOwL3kFnpE8T5fbUpPUV4+X/8 x0FIxUUaOkCEP5FMdfIOl8H2zUB1ODjKYurLZJxgp/OClWbg9ZWyVlcC+4qGU3KfEyUi VyKoSVlZHSjH1bUuqdKGZZZZ/WVT++WfTOwMoLQY4Dj8W6+e7/3YaNyMqYM2mh01EAL5 iPtPt7GEmeEn2T45RO6Vesu1Uskl24JQtFhNI8oQYme+7VMxLcW7aDInumjRUtYyAGmS lMYJTbANBZ2/g65fcSTN7byqMOpROtAneO0022XnXWJlbdRnZeGmOLhaG1qaqC8Hg796 gFJA== X-Gm-Message-State: AOAM531SEOxZerr89zO1dB3WY5sr81MNX4DqutIn7Iny8XEVpbR2JnFF lA8te6dlS3FqinfuEN8dCEao/vARIeO3gw== X-Google-Smtp-Source: ABdhPJzFqvfJge1D0buCJNK6Nuo7zl6pHQuJxOm47k7+EI6EZYE3vnRc3ux4rSTPHFk4tq2uaaLx0w== X-Received: by 2002:a17:906:3b56:: with SMTP id h22mr52359670ejf.491.1609427864869; Thu, 31 Dec 2020 07:17:44 -0800 (PST) Received: from localhost.localdomain ([31.45.254.141]) by smtp.gmail.com with ESMTPSA id hb18sm20087509ejb.86.2020.12.31.07.17.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Dec 2020 07:17:44 -0800 (PST) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Thu, 31 Dec 2020 16:17:36 +0100 Message-Id: <20201231151736.12042-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avfilter: add estdif video filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Signed-off-by: Paul B Mahol --- TODO: add working >8 depth support --- doc/filters.texi | 56 ++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_estdif.c | 372 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 430 insertions(+) create mode 100644 libavfilter/vf_estdif.c diff --git a/doc/filters.texi b/doc/filters.texi index 01ae540c5c..d5cfb1b4ff 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -11065,6 +11065,62 @@ Flags to local 3x3 coordinates maps like this: This filter supports the all above options as @ref{commands}. +@section estdif + +Deinterlace the input video ("estdif" stands for "Edge Slope +Tracing Deinterlacing Filter"). + +Spatial only filter that uses simple edge slope tracing algorithm +to interpolate missing lines. +It accepts the following parameters: + +@table @option +@item mode +The interlacing mode to adopt. It accepts one of the following values: + +@table @option +@item frame +Output one frame for each frame. +@item field +Output one frame for each field. +@end table + +The default value is @code{field}. + +@item parity +The picture field parity assumed for the input interlaced video. It accepts one +of the following values: + +@table @option +@item ff +Assume the top field is first. +@item bff +Assume the bottom field is first. +@item auto +Enable automatic detection of field parity. +@end table + +The default value is @code{auto}. +If the interlacing is unknown or the decoder does not export this information, +top field first will be assumed. + +@item deint +Specify which frames to deinterlace. Accepts one of the following +values: + +@table @option +@item all +Deinterlace all frames. +@item interlaced +Only deinterlace frames marked as interlaced. +@end table + +The default value is @code{all}. +@end table + +@subsection Commands +This filter supports same @ref{commands} as options. + @section extractplanes Extract color channel components from input video stream into diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 526da8d69e..7939381616 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -252,6 +252,7 @@ OBJS-$(CONFIG_EQ_FILTER) += vf_eq.o OBJS-$(CONFIG_EROSION_FILTER) += vf_neighbor.o OBJS-$(CONFIG_EROSION_OPENCL_FILTER) += vf_neighbor_opencl.o opencl.o \ opencl/neighbor.o +OBJS-$(CONFIG_ESTDIF_FILTER) += vf_estdif.o OBJS-$(CONFIG_EXTRACTPLANES_FILTER) += vf_extractplanes.o OBJS-$(CONFIG_FADE_FILTER) += vf_fade.o OBJS-$(CONFIG_FFTDNOIZ_FILTER) += vf_fftdnoiz.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index ce317dfa1c..471844a603 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -237,6 +237,7 @@ extern AVFilter ff_vf_entropy; extern AVFilter ff_vf_eq; extern AVFilter ff_vf_erosion; extern AVFilter ff_vf_erosion_opencl; +extern AVFilter ff_vf_estdif; extern AVFilter ff_vf_extractplanes; extern AVFilter ff_vf_fade; extern AVFilter ff_vf_fftdnoiz; diff --git a/libavfilter/vf_estdif.c b/libavfilter/vf_estdif.c new file mode 100644 index 0000000000..ec2cd42a96 --- /dev/null +++ b/libavfilter/vf_estdif.c @@ -0,0 +1,372 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" +#include "libavutil/imgutils.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" +#include "avfilter.h" +#include "formats.h" +#include "internal.h" +#include "video.h" + +typedef struct ESTDIFContext { + const AVClass *class; + + int mode; ///< 0 is frame, 1 is field + int parity; ///< frame field parity + int deint; ///< which frames to deinterlace + int linesize[4]; ///< bytes of pixel data per line for each plane + int planewidth[4]; ///< width of each plane + int planeheight[4]; ///< height of each plane + int field; ///< which field are we on, 0 or 1 + int eof; + int nb_planes; + int nb_threads; + int max; + int64_t pts; + AVFrame *prev; + + int *work_line[4]; +} ESTDIFContext; + +#define OFFSET(x) offsetof(ESTDIFContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM|AV_OPT_FLAG_RUNTIME_PARAM +#define CONST(name, help, val, unit) { name, help, 0, AV_OPT_TYPE_CONST, {.i64=val}, 0, 0, FLAGS, unit } + +static const AVOption estdif_options[] = { + { "mode", "specify the mode", OFFSET(mode), AV_OPT_TYPE_INT, {.i64=1}, 0, 1, FLAGS, "mode" }, + CONST("frame", NULL, 0, "mode"), + CONST("field", NULL, 1, "mode"), + { "parity", "specify the assumed picture field parity", OFFSET(parity), AV_OPT_TYPE_INT, {.i64=-1}, -1, 1, FLAGS, "parity" }, + CONST("tff", "assume top field first", 0, "parity"), + CONST("bff", "assume bottom field first", 1, "parity"), + CONST("auto", "auto detect parity", -1, "parity"), + { "deint", "specify which frames to deinterlace", OFFSET(deint), AV_OPT_TYPE_INT, {.i64=0}, 0, 1, FLAGS, "deint" }, + CONST("all", "deinterlace all frames", 0, "deint"), + CONST("interlaced", "only deinterlace frames marked as interlaced", 1, "deint"), + { NULL } +}; + +AVFILTER_DEFINE_CLASS(estdif); + +static int query_formats(AVFilterContext *ctx) +{ + static const enum AVPixelFormat pix_fmts[] = { + AV_PIX_FMT_YUV410P, AV_PIX_FMT_YUV411P, + AV_PIX_FMT_YUV420P, AV_PIX_FMT_YUV422P, + AV_PIX_FMT_YUV440P, AV_PIX_FMT_YUV444P, + AV_PIX_FMT_YUVJ444P, AV_PIX_FMT_YUVJ440P, + AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ420P, + AV_PIX_FMT_YUVJ411P, + AV_PIX_FMT_YUVA420P, AV_PIX_FMT_YUVA422P, AV_PIX_FMT_YUVA444P, + AV_PIX_FMT_GBRP, AV_PIX_FMT_GBRAP, + AV_PIX_FMT_GRAY8, + AV_PIX_FMT_YUV420P9, AV_PIX_FMT_YUV422P9, AV_PIX_FMT_YUV444P9, + AV_PIX_FMT_YUV420P10, AV_PIX_FMT_YUV422P10, AV_PIX_FMT_YUV444P10, + AV_PIX_FMT_YUV420P12, AV_PIX_FMT_YUV422P12, AV_PIX_FMT_YUV444P12, + AV_PIX_FMT_YUV420P14, AV_PIX_FMT_YUV422P14, AV_PIX_FMT_YUV444P14, + AV_PIX_FMT_GBRP9, AV_PIX_FMT_GBRP10, AV_PIX_FMT_GBRP12, AV_PIX_FMT_GBRP14, + AV_PIX_FMT_YUVA444P9, AV_PIX_FMT_YUVA444P10, AV_PIX_FMT_YUVA444P12, AV_PIX_FMT_YUVA444P16, + AV_PIX_FMT_YUVA422P9, AV_PIX_FMT_YUVA422P10, AV_PIX_FMT_YUVA422P12, AV_PIX_FMT_YUVA422P16, + AV_PIX_FMT_YUVA420P9, AV_PIX_FMT_YUVA420P10, AV_PIX_FMT_YUVA420P16, + AV_PIX_FMT_GBRAP10, AV_PIX_FMT_GBRAP12, AV_PIX_FMT_GBRAP16, + AV_PIX_FMT_NONE + }; + + AVFilterFormats *fmts_list = ff_make_format_list(pix_fmts); + if (!fmts_list) + return AVERROR(ENOMEM); + return ff_set_common_formats(ctx, fmts_list); +} + +static int config_input(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + ESTDIFContext *s = ctx->priv; + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(inlink->format); + int ret, depth; + + if ((ret = av_image_fill_linesizes(s->linesize, inlink->format, inlink->w)) < 0) + return ret; + + s->planeheight[1] = s->planeheight[2] = AV_CEIL_RSHIFT(inlink->h, desc->log2_chroma_h); + s->planeheight[0] = s->planeheight[3] = inlink->h; + s->planewidth[1] = s->planewidth[2] = AV_CEIL_RSHIFT(inlink->w, desc->log2_chroma_w); + s->planewidth[0] = s->planewidth[3] = inlink->w; + + if (inlink->h < 3) { + av_log(ctx, AV_LOG_ERROR, "Video of less than 3 lines is not supported\n"); + return AVERROR(EINVAL); + } + + s->nb_planes = av_pix_fmt_count_planes(inlink->format); + s->nb_threads = ff_filter_get_nb_threads(ctx); + depth = desc->comp[0].depth; + s->max = (1 << depth) - 1; + + for (int plane = 0; plane < s->nb_planes; plane++) { + s->work_line[plane] = av_calloc(FFALIGN(s->planewidth[plane], 32) * s->nb_threads * 3, sizeof(*s->work_line[0])); + if (!s->work_line[plane]) + return AVERROR(ENOMEM); + } + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = ctx->inputs[0]; + + outlink->time_base.num = inlink->time_base.num; + outlink->time_base.den = inlink->time_base.den * 2; + outlink->frame_rate.num = inlink->frame_rate.num * 2; + outlink->frame_rate.den = inlink->frame_rate.den; + + return 0; +} + +typedef struct ThreadData { + AVFrame *out, *in; +} ThreadData; + +static void interpolate_dir(int *work_line, int *la, + uint8_t *pprev_line, uint8_t *prev_line, + uint8_t *next_line, int x, int width, + int first, int *K, int *SMIN) +{ + const int px = FFMAX(0, x - 1); + const int nx = FFMIN(width - 1, x); + int d1 = FFABS(prev_line[px] - next_line[px]) + + FFABS(prev_line[x ] - next_line[x ]) + + FFABS(prev_line[nx] - next_line[nx]); + int d2 = FFABS(prev_line[px] - next_line[x ]) + + FFABS(prev_line[x ] - next_line[nx]); + int d3 = FFABS(prev_line[x ] - next_line[px]) + + FFABS(prev_line[nx] - next_line[x ]); + int k = *K; + int sprev = *SMIN; + + if (FFMIN3(d1, d2, d3) < 20) { + work_line[x] = la[x]; + k = 0; + } else { + int s0 = FFABS(prev_line[av_clip(x + k - 1, first, width - 1)] - next_line[av_clip(x - k + 1, first, width - 1)]); + int s1 = FFABS(prev_line[av_clip(x + k , first, width - 1)] - next_line[av_clip(x - k , first, width - 1)]); + int s2 = FFABS(prev_line[av_clip(x + k + 1, first, width - 1)] - next_line[av_clip(x - k - 1, first, width - 1)]); + int v, smin = FFMIN3(s0, s1, s2); + + if (smin == s2) + k--; + else if (smin == s0) + k++; + + if (FFABS(smin - sprev) > 10 && FFABS(k) > 1) + k = 0; + + sprev = smin; + v = (prev_line[av_clip(x - k, 0, width - 1)] + next_line[av_clip(x + k, 0, width - 1)]) >> 1; + work_line[x] = v; + } + + *K = k; + *SMIN = sprev; +} + +static int deinterlace_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) +{ + ESTDIFContext *s = ctx->priv; + ThreadData *td = arg; + AVFrame *out = td->out; + AVFrame *in = td->in; + const int field_first = s->parity == -1 ? !in->top_field_first : s->parity == 0 ? 1 : 0; + + for (int plane = 0; plane < s->nb_planes; plane++) { + uint8_t *src_data = in->data[plane]; + uint8_t *dst_data = out->data[plane]; + const int linesize = s->linesize[plane]; + const int width = s->planewidth[plane]; + const int height = s->planeheight[plane]; + const int src_linesize = in->linesize[plane]; + const int dst_linesize = out->linesize[plane]; + const int start = (((height / 2) * jobnr) / nb_jobs) * 2; + const int end = (((height / 2) * (jobnr+1)) / nb_jobs) * 2; + int *work_line0 = s->work_line[plane] + jobnr * 3 * width + 0 * width; + int *work_line1 = s->work_line[plane] + jobnr * 3 * width + 1 * width; + int *work_line2 = s->work_line[plane] + jobnr * 3 * width + 2 * width; + uint8_t *pprev_line, *prev_line, *next_line, *in_line; + uint8_t *out_line; + const int first = ((s->field == field_first) ^ (start & 1)); + int y_out; + + y_out = start + ((s->field == field_first) ^ (start & 1)); + + in_line = src_data + (y_out * src_linesize); + out_line = dst_data + (y_out * dst_linesize); + + while (y_out < end) { + memcpy(out_line, in_line, linesize); + y_out += 2; + in_line += src_linesize * 2; + out_line += dst_linesize * 2; + } + + y_out = start + ((s->field != field_first) ^ (start & 1)); + out_line = dst_data + (y_out * dst_linesize); + + for (int y = y_out; y < end; y += 2) { + int k, smin; + + pprev_line = src_data + (FFMAX(first, y - 2) * src_linesize); + prev_line = src_data + (FFMAX(first, y - 1) * src_linesize); + next_line = src_data + (FFMIN(height - 1, y + 1) * src_linesize); + + k = 0, smin = 0; + for (int x = 0; x < width; x++) { + work_line2[x] = (prev_line[x] + next_line[x]) >> 1; + } + + for (int x = 0; x < width; x++) { + interpolate_dir(work_line0, work_line2, + pprev_line, prev_line, next_line, + x, width, first, &k, &smin); + } + + k = 0, smin = 0; + for (int x = width - 1; x >= 0; x--) { + interpolate_dir(work_line1, work_line2, + pprev_line, prev_line, next_line, + x, width, first, &k, &smin); + } + + for (int x = 0; x < width; x++) { + int dlr = FFABS(work_line0[x] - work_line2[x]); + int drl = FFABS(work_line1[x] - work_line2[x]); + + if (dlr < drl) + out_line[x] = work_line0[x]; + else + out_line[x] = work_line1[x]; + } + + out_line += 2 * dst_linesize; + } + } + + return 0; +} + +static int filter(AVFilterContext *ctx, int is_second, AVFrame *in) +{ + ESTDIFContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + AVFrame *out; + ThreadData td; + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) + return AVERROR(ENOMEM); + av_frame_copy_props(out, in); + out->interlaced_frame = 0; + out->pts = s->pts; + + td.out = out; td.in = in; + ctx->internal->execute(ctx, deinterlace_slice, &td, NULL, + FFMIN(s->planeheight[1] / 2, s->nb_threads)); + + if (s->mode) + s->field = !s->field; + + return ff_filter_frame(outlink, out); +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + AVFilterContext *ctx = inlink->dst; + ESTDIFContext *s = ctx->priv; + int ret; + + if (!s->prev) { + s->prev = in; + return 0; + } + + if ((s->deint && !in->interlaced_frame) || ctx->is_disabled) { + s->prev->pts *= 2; + ret = ff_filter_frame(ctx->outputs[0], s->prev); + s->prev = in; + return ret; + } + + s->pts = s->prev->pts * 2; + ret = filter(ctx, 0, s->prev); + if (ret < 0 || s->mode == 0) { + av_frame_free(&s->prev); + s->prev = in; + return ret; + } + + s->pts = s->prev->pts + in->pts; + ret = filter(ctx, 1, s->prev); + av_frame_free(&s->prev); + s->prev = in; + return ret; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + ESTDIFContext *s = ctx->priv; + + av_freep(&s->prev); + + for (int i = 0; i < s->nb_planes; i++) + av_freep(&s->work_line[i]); +} + +static const AVFilterPad estdif_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + .config_props = config_input, + }, + { NULL } +}; + +static const AVFilterPad estdif_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + }, + { NULL } +}; + +AVFilter ff_vf_estdif = { + .name = "estdif", + .description = NULL_IF_CONFIG_SMALL("Apply Edge Slope Tracing deinterlace."), + .priv_size = sizeof(ESTDIFContext), + .priv_class = &estdif_class, + .uninit = uninit, + .query_formats = query_formats, + .inputs = estdif_inputs, + .outputs = estdif_outputs, + .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_INTERNAL | AVFILTER_FLAG_SLICE_THREADS, + .process_command = ff_filter_process_command, +};