From patchwork Wed Apr 24 10:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 48248 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a21:2d06:b0:1a9:af23:56c1 with SMTP id tw6csp571156pzb; Wed, 24 Apr 2024 04:03:45 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXAD0uRQ7VU55x7dhrqk6glARvy+WDU+QdqmLyMmvaOs0lxmG7XsoECj+xNErLajrxbPm4u5c5U6AhKY9EOZaE9pQyfRe2C2cmkYA== X-Google-Smtp-Source: AGHT+IE6zIsBRnEBMmaQ6VQtIWnbsEI5PnFpjb4h0ti3OD0BbbYkV6JdlcUVl5zAsJbdm8mhtwXa X-Received: by 2002:a17:907:7e86:b0:a55:784c:858c with SMTP id qb6-20020a1709077e8600b00a55784c858cmr1506951ejc.4.1713956624936; Wed, 24 Apr 2024 04:03:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1713956624; cv=none; d=google.com; s=arc-20160816; b=ppEklFSSBB2LktmGfDQA9YYszYTW4yJqTz5hBpyV0boJbsmb2oOADRgUQtmAh/kmfX eam0bc2hhrMXDLslzGHXj1V4OH0tbW+1AGRUWcDvOfQWAuG4JBZQ7be1Hg7doyeh0Tsl wGWpeSX5iVW8lIxd7L6medO+DNMpXV7IFftAbOk8JEM0aTzqgHAwEGAclJdmN22QUdd4 Bp9knX+InR8L/qs94gmaBAM5aKaD/ItD0GfRIihEITHu/dD2cuBEitvZcvWZ+4Virrn5 hMoBiFfgE+Fu1wdJu++H1HkYyiItHRoaNtMOf+Pk5HzunUsDUTic/ITajn7eXcMYOHWj peIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=s+0hA9p9xqy9q+l9MyOGSi+i//rgvoLuGP+5so21ZR4=; fh=xmAeKtysnShNOmkhiJmYkS30uw4Fu2hvBJ7qlIwukxQ=; b=OvvUz2CQtZ/OihRGJ/9ql7kajPEzySwGAtx+QtWm0duQGyAuqrErdQDYvfdjoK2ijr aBtwlnA6AwnhiRiUhY7v6n/Os6RbQ6xDug2ZXHGUk2F8BovxIevHUJrdpUQ8vic3d88P S4do0VyDdylDIIYG0gx4hTgESMcEw9RM6dowSvTZ6jXimaqlecZDOJruHPViuQLDkjKB bNRFJEkv3FgE/8IdY580bxfgWd+q5gqqniH8Rw1uLikMmL2u7C1TBw02GdFwhbUB7nMI bo71fx6wGeqyBulebwPj3VSL/Yv6iKu+3xACOzOOqKqpg40RT6/uesKufbWmN7ZGTdMe 780A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=HluKfG5D; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gb32-20020a170907962000b00a55b1e098fbsi4636894ejc.45.2024.04.24.04.03.43; Wed, 24 Apr 2024 04:03:44 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=HluKfG5D; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1BD9668D2B5; Wed, 24 Apr 2024 14:03:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id D957068D0D9 for ; Wed, 24 Apr 2024 14:03:05 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1713956585; bh=4OCt+2VlvWufavFb0q6+P7KP6bxXBcAzx2Oy9Fnee14=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HluKfG5DVq9SX3pEiqrWLDScwwWtzMIrPIda5qQLzOVA6tECA/EYKTchFQvq52xSt H0pI0wvp8TSAme4Y3zhvSzhxSvUJfFIf1yL8kRCiwEvcBRnN4AD67P/THgmvch6K+t hp0RydW5R9lf2G9CELTSa/AzjW4BvDOhHaXOxpNk= Received: from haasn.dev (unknown [10.30.0.2]) by haasn.dev (Postfix) with ESMTP id 9BB9442922; Wed, 24 Apr 2024 13:03:05 +0200 (CEST) From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 24 Apr 2024 12:51:58 +0200 Message-ID: <20240424110257.38715-4-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424110257.38715-1-ffmpeg@haasn.xyz> References: <20240424110257.38715-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/5] avfilter/vf_scale: add optional "ref" input X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: DrN0dn+htq98 From: Niklas Haas This is automatically enabled if the width/height expressions reference any ref_* variable. This will ultimately serve as a more principled replacement for the fundamentally broken scale2ref. See-Also: https://trac.ffmpeg.org/ticket/10795 --- Changelog | 1 + doc/filters.texi | 26 ++++++++ libavfilter/vf_scale.c | 132 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 156 insertions(+), 3 deletions(-) diff --git a/Changelog b/Changelog index 8db14f02b4..e821e5ac74 100644 --- a/Changelog +++ b/Changelog @@ -7,6 +7,7 @@ version : - ffmpeg CLI filtergraph chaining - LC3/LC3plus demuxer and muxer - pad_vaapi, drawbox_vaapi filters +- vf_scale supports secondary ref input and framesync options version 7.0: diff --git a/doc/filters.texi b/doc/filters.texi index f20b72ab96..cf884568b0 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -21562,8 +21562,34 @@ The position (byte offset) of the frame in the input stream, or NaN if this information is unavailable and/or meaningless (for example in case of synthetic video). Only available with @code{eval=frame}. Deprecated, do not use. + +@item ref_w, rw +@item ref_h, rh +@item ref_a +@item ref_dar, rdar +@item ref_n +@item ref_t +@item ref_pos +Eqvuialent to the above, but for a second reference input. If any of these +variables are present, this filter accepts two inputs. @end table +@subsection Examples + +@itemize +@item +Scale a subtitle stream (sub) to match the main video (main) in size before overlaying +@example +'[main]split[a][b]; [ref][a]scale=rw:rh[c]; [b][c]overlay' +@end example + +@item +Scale a logo to 1/10th the height of a video, while preserving its display aspect ratio. +@example +[logo-in][video-in]scale=w=oh*dar:h=rh/10[logo-out] +@end example +@end itemize + @section scale2ref Scale (resize) the input video, based on a reference video. diff --git a/libavfilter/vf_scale.c b/libavfilter/vf_scale.c index a986dc97ae..f174651333 100644 --- a/libavfilter/vf_scale.c +++ b/libavfilter/vf_scale.c @@ -59,6 +59,17 @@ static const char *const var_names[] = { #if FF_API_FRAME_PKT "pos", #endif + "ref_w", "rw", + "ref_h", "rh", + "ref_a", + "ref_sar", + "ref_dar", "rdar", + "ref_hsub", + "ref_vsub", + "ref_n", + "ref_t", + "ref_pos", + /* Legacy variables for scale2ref */ "main_w", "main_h", "main_a", @@ -89,6 +100,16 @@ enum var_name { #if FF_API_FRAME_PKT VAR_POS, #endif + VAR_REF_W, VAR_RW, + VAR_REF_H, VAR_RH, + VAR_REF_A, + VAR_REF_SAR, + VAR_REF_DAR, VAR_RDAR, + VAR_REF_HSUB, + VAR_REF_VSUB, + VAR_REF_N, + VAR_REF_T, + VAR_REF_POS, VAR_S2R_MAIN_W, VAR_S2R_MAIN_H, VAR_S2R_MAIN_A, @@ -131,6 +152,7 @@ typedef struct ScaleContext { int input_is_pal; ///< set to 1 if the input format is paletted int output_is_pal; ///< set to 1 if the output format is paletted int interlaced; + int uses_ref; char *w_expr; ///< width expression string char *h_expr; ///< height expression string @@ -190,6 +212,38 @@ static int check_exprs(AVFilterContext *ctx) av_log(ctx, AV_LOG_WARNING, "Circular references detected for width '%s' and height '%s' - possibly invalid.\n", scale->w_expr, scale->h_expr); } + if (vars_w[VAR_REF_W] || vars_h[VAR_REF_W] || + vars_w[VAR_RW] || vars_h[VAR_RW] || + vars_w[VAR_REF_H] || vars_h[VAR_REF_H] || + vars_w[VAR_RH] || vars_h[VAR_RH] || + vars_w[VAR_REF_A] || vars_h[VAR_REF_A] || + vars_w[VAR_REF_SAR] || vars_h[VAR_REF_SAR] || + vars_w[VAR_REF_DAR] || vars_h[VAR_REF_DAR] || + vars_w[VAR_RDAR] || vars_h[VAR_RDAR] || + vars_w[VAR_REF_HSUB] || vars_h[VAR_REF_HSUB] || + vars_w[VAR_REF_VSUB] || vars_h[VAR_REF_VSUB] || + vars_w[VAR_REF_N] || vars_h[VAR_REF_N] || + vars_w[VAR_REF_T] || vars_h[VAR_REF_T] || + vars_w[VAR_REF_POS] || vars_h[VAR_REF_POS]) { + scale->uses_ref = 1; + } + + if (ctx->filter != &ff_vf_scale2ref && + (vars_w[VAR_S2R_MAIN_W] || vars_h[VAR_S2R_MAIN_W] || + vars_w[VAR_S2R_MAIN_H] || vars_h[VAR_S2R_MAIN_H] || + vars_w[VAR_S2R_MAIN_A] || vars_h[VAR_S2R_MAIN_A] || + vars_w[VAR_S2R_MAIN_SAR] || vars_h[VAR_S2R_MAIN_SAR] || + vars_w[VAR_S2R_MAIN_DAR] || vars_h[VAR_S2R_MAIN_DAR] || + vars_w[VAR_S2R_MDAR] || vars_h[VAR_S2R_MDAR] || + vars_w[VAR_S2R_MAIN_HSUB] || vars_h[VAR_S2R_MAIN_HSUB] || + vars_w[VAR_S2R_MAIN_VSUB] || vars_h[VAR_S2R_MAIN_VSUB] || + vars_w[VAR_S2R_MAIN_N] || vars_h[VAR_S2R_MAIN_N] || + vars_w[VAR_S2R_MAIN_T] || vars_h[VAR_S2R_MAIN_T] || + vars_w[VAR_S2R_MAIN_POS] || vars_h[VAR_S2R_MAIN_POS]) ) { + av_log(ctx, AV_LOG_ERROR, "Expressions with scale2ref variables are not valid in scale filter.\n"); + return AVERROR(EINVAL); + } + if (ctx->filter != &ff_vf_scale2ref && (vars_w[VAR_S2R_MAIN_W] || vars_h[VAR_S2R_MAIN_W] || vars_w[VAR_S2R_MAIN_H] || vars_h[VAR_S2R_MAIN_H] || @@ -385,6 +439,9 @@ static av_cold int init(AVFilterContext *ctx) if (!threads) av_opt_set_int(scale->sws_opts, "threads", ff_filter_get_nb_threads(ctx), 0); + if (ctx->filter != &ff_vf_scale2ref) + ctx->nb_inputs = scale->uses_ref ? 2 : 1; + return 0; } @@ -506,6 +563,20 @@ static int scale_eval_dimensions(AVFilterContext *ctx) scale->var_values[VAR_S2R_MAIN_VSUB] = 1 << main_desc->log2_chroma_h; } + if (scale->uses_ref) { + const AVFilterLink *reflink = ctx->inputs[1]; + const AVPixFmtDescriptor *ref_desc = av_pix_fmt_desc_get(reflink->format); + scale->var_values[VAR_REF_W] = scale->var_values[VAR_RW] = reflink->w; + scale->var_values[VAR_REF_H] = scale->var_values[VAR_RH] = reflink->h; + scale->var_values[VAR_REF_A] = (double) reflink->w / reflink->h; + scale->var_values[VAR_REF_SAR] = reflink->sample_aspect_ratio.num ? + (double) reflink->sample_aspect_ratio.num / reflink->sample_aspect_ratio.den : 1; + scale->var_values[VAR_REF_DAR] = scale->var_values[VAR_RDAR] = + scale->var_values[VAR_REF_A] * scale->var_values[VAR_REF_SAR]; + scale->var_values[VAR_REF_HSUB] = 1 << ref_desc->log2_chroma_w; + scale->var_values[VAR_REF_VSUB] = 1 << ref_desc->log2_chroma_h; + } + res = av_expr_eval(scale->w_pexpr, scale->var_values, NULL); eval_w = scale->var_values[VAR_OUT_W] = scale->var_values[VAR_OW] = (int) res == 0 ? inlink->w : (int) res; @@ -693,6 +764,13 @@ static int config_props(AVFilterLink *outlink) scale->fs.in[0].sync = 1; scale->fs.in[0].before = EXT_STOP; scale->fs.in[0].after = EXT_STOP; + if (scale->uses_ref) { + av_assert0(ctx->nb_inputs == 2); + scale->fs.in[1].time_base = ctx->inputs[1]->time_base; + scale->fs.in[1].sync = 0; + scale->fs.in[1].before = EXT_NULL; + scale->fs.in[1].after = EXT_INFINITY; + } ret = ff_framesync_configure(&scale->fs); if (ret < 0) @@ -919,13 +997,55 @@ scale: static int do_scale(FFFrameSync *fs) { AVFilterContext *ctx = fs->parent; + ScaleContext *scale = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; - AVFrame *in, *out; - int ret; + AVFrame *out, *in = NULL, *ref = NULL; + int ret = 0, frame_changed; ret = ff_framesync_get_frame(fs, 0, &in, 1); if (ret < 0) - return ret; + goto err; + + if (scale->uses_ref) { + ret = ff_framesync_get_frame(fs, 1, &ref, 0); + if (ret < 0) + goto err; + } + + if (ref) { + AVFilterLink *reflink = ctx->inputs[1]; + frame_changed = ref->width != reflink->w || + ref->height != reflink->h || + ref->format != reflink->format || + ref->sample_aspect_ratio.den != reflink->sample_aspect_ratio.den || + ref->sample_aspect_ratio.num != reflink->sample_aspect_ratio.num || + ref->colorspace != reflink->colorspace || + ref->color_range != reflink->color_range; + + if (frame_changed) { + reflink->format = ref->format; + reflink->w = ref->width; + reflink->h = ref->height; + reflink->sample_aspect_ratio.num = ref->sample_aspect_ratio.num; + reflink->sample_aspect_ratio.den = ref->sample_aspect_ratio.den; + reflink->colorspace = ref->colorspace; + reflink->color_range = ref->color_range; + + ret = config_props(outlink); + if (ret < 0) + goto err; + } + + if (scale->eval_mode == EVAL_MODE_FRAME) { + scale->var_values[VAR_REF_N] = reflink->frame_count_out; + scale->var_values[VAR_REF_T] = TS2T(ref->pts, reflink->time_base); +#if FF_API_FRAME_PKT +FF_DISABLE_DEPRECATION_WARNINGS + scale->var_values[VAR_REF_POS] = ref->pkt_pos == -1 ? NAN : ref->pkt_pos; +FF_ENABLE_DEPRECATION_WARNINGS +#endif + } + } ret = scale_frame(ctx->inputs[0], in, &out); if (out) { @@ -933,6 +1053,9 @@ static int do_scale(FFFrameSync *fs) return ff_filter_frame(outlink, out); } +err: + if (ret < 0) + av_frame_free(&in); return ret; } @@ -1108,6 +1231,9 @@ static const AVFilterPad avfilter_vf_scale_inputs[] = { { .name = "default", .type = AVMEDIA_TYPE_VIDEO, + }, { + .name = "ref", + .type = AVMEDIA_TYPE_VIDEO, }, };