From patchwork Wed May 10 13:55:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41566 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867493pzb; Wed, 10 May 2023 06:55:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4WYZeEL1xO29ST1e3jEhbRtwBOoooJBwpNGnpxXHzF4nUDjPqEQ6oIibs6ZW3nwdwFDIj1 X-Received: by 2002:a17:907:9345:b0:94e:c4b:4d95 with SMTP id bv5-20020a170907934500b0094e0c4b4d95mr14734595ejc.69.1683726943741; Wed, 10 May 2023 06:55:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726943; cv=none; d=google.com; s=arc-20160816; b=lFY4180z6XwKHfL/AnVdsUYkRMvNz6N7pT9ldQiisAX1wDUp4k3K6j1D8Nfu0hTfW0 /BN4HZo19r9plZvU1i8clvaeD1qi1rC/tsnhGXT9vfqFSRlY+JrPINgVLMzwKzgaKsYU YHqfJaBfxFMKneJliZ2NOmEAy4wlnmzSyts9Kg90ixLl+bkpfoeVyDQKy0t/ud4h1E3t wQPPtieYAy7hCL1rew/58VGSOmemuvHm7J0wL4iryupb7IgFiByKLkdKLOYLWrDV0RAK 6YhenCFzSVjs+VV74S7ChwVPAn/YMvuMaqpVOuyUneCHBUSDAr6m6AXTbU4EBYpKlccv 8w8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=Q2xEqLCS6ZvLzOzk50Y3nFNwzo4bHsw/Km167TkBHTE=; b=1HMCeM7DC032sCvzjQK2R3EtPHv3kwCWrag5yQjGvuAj76tBC09CWwFy7Njjqh9B4O 84SAFJ1ovHsCsSLmK99XlrJisfeVIvog4rzzAAZ6dGVozW/f+2AiXxdr+77/H5vGV378 kkdrZkRk1Cv76rmb8z9WFhog62DEfXLPMsYEIj/MjpFZTEt7066inaQq5+1BYiDr6Juw JIqEZtkGzl3Tti1apehveaErZQIPeHKgcnosGpT96uSijw1lYgFzxw1PxKiYye5k9xjI 1rExB3CRRdE5qvmT5UxyKQBvXPQ7yPYL7rwGEwu30Dv3amWPNvxpEGAnM303kh5mZ5Bv 5wPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="tm/R1UQ9"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id tl5-20020a170907c30500b0096632ad29desi3122759ejc.685.2023.05.10.06.55.43; Wed, 10 May 2023 06:55:43 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b="tm/R1UQ9"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6C55C68C138; Wed, 10 May 2023 16:55:25 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1F49A68A906 for ; Wed, 10 May 2023 16:55:17 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id 5951B40E98; Wed, 10 May 2023 15:55:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726915; bh=o7P878m4O3PFDZmn4r67wfVZbiLgQgNE16XnztPJ4cE=; h=From:To:Cc:Subject:Date:From; b=tm/R1UQ9zJ1pmVskfoYeIv6Wg39PUX7dqEqm7TgV+WE8Y0JOyXsCljq5ceHjauOsU imumpmGTsHN00H2v+Pj1859S6aidpFEwIyT88quQSh2VqZ22Yvc/la4X48XGaLrtpK f/CyKtuIgAWscfSWrMoJCW4VgzWyyuAt7T9YE6mI= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:04 +0200 Message-Id: <20230510135509.4074-1-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/6] lavfi/vf_libplacebo: update render params on demand X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: P+4lCKXI9JYT From: Niklas Haas Only update this struct when it's expected to change, and cache it otherwise. Partially motivated by a desire to make `process_frames` smaller. --- libavfilter/vf_libplacebo.c | 228 ++++++++++++++++++++---------------- 1 file changed, 126 insertions(+), 102 deletions(-) diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index 6fe3e0ea88..2ee9055c9d 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -143,6 +143,7 @@ typedef struct LibplaceboContext { int color_trc; /* pl_render_params */ + struct pl_render_params params; char *upscaler; char *downscaler; int lut_entries; @@ -156,6 +157,7 @@ typedef struct LibplaceboContext { int disable_fbos; /* pl_deband_params */ + struct pl_deband_params deband_params; int deband; int deband_iterations; float deband_threshold; @@ -163,6 +165,7 @@ typedef struct LibplaceboContext { float deband_grain; /* pl_color_adjustment */ + struct pl_color_adjustment color_adjustment; float brightness; float contrast; float saturation; @@ -170,6 +173,7 @@ typedef struct LibplaceboContext { float gamma; /* pl_peak_detect_params */ + struct pl_peak_detect_params peak_detect_params; int peakdetect; float smoothing; float min_peak; @@ -178,6 +182,7 @@ typedef struct LibplaceboContext { float overshoot; /* pl_color_map_params */ + struct pl_color_map_params color_map_params; int intent; int gamut_mode; int tonemapping; @@ -196,12 +201,14 @@ typedef struct LibplaceboContext { int force_icc_lut; #endif - /* pl_dither_params */ + /* pl_dither_params */ + struct pl_dither_params dither_params; int dithering; int dither_lut_size; int dither_temporal; /* pl_cone_params */ + struct pl_cone_params cone_params; int cones; float cone_str; @@ -280,18 +287,112 @@ static int find_scaler(AVFilterContext *avctx, return AVERROR(EINVAL); } -static int parse_fillcolor(AVFilterContext *avctx, - struct pl_render_params *params, - const char *color_str) +static int update_settings(AVFilterContext *ctx) { int err = 0; + LibplaceboContext *s = ctx->priv; + enum pl_tone_map_mode tonemapping_mode = s->tonemapping_mode; + enum pl_gamut_mode gamut_mode = s->gamut_mode; uint8_t color_rgba[4]; - RET(av_parse_color(color_rgba, color_str, -1, avctx)); - params->background_color[0] = (float) color_rgba[0] / UINT8_MAX; - params->background_color[1] = (float) color_rgba[1] / UINT8_MAX; - params->background_color[2] = (float) color_rgba[2] / UINT8_MAX; - params->background_transparency = 1.0f - (float) color_rgba[3] / UINT8_MAX; + RET(av_parse_color(color_rgba, s->fillcolor, -1, s)); + +#if FF_API_LIBPLACEBO_OPTS + /* backwards compatibility with older API */ + if (!tonemapping_mode && (s->desat_str >= 0.0f || s->desat_exp >= 0.0f)) { + float str = s->desat_str < 0.0f ? 0.9f : s->desat_str; + float exp = s->desat_exp < 0.0f ? 0.2f : s->desat_exp; + if (str >= 0.9f && exp <= 0.1f) { + tonemapping_mode = PL_TONE_MAP_RGB; + } else if (str > 0.1f) { + tonemapping_mode = PL_TONE_MAP_HYBRID; + } else { + tonemapping_mode = PL_TONE_MAP_LUMA; + } + } + + if (s->gamut_warning) + gamut_mode = PL_GAMUT_WARN; + if (s->gamut_clipping) + gamut_mode = PL_GAMUT_DESATURATE; +#endif + + s->deband_params = *pl_deband_params( + .iterations = s->deband_iterations, + .threshold = s->deband_threshold, + .radius = s->deband_radius, + .grain = s->deband_grain, + ); + + s->color_adjustment = (struct pl_color_adjustment) { + .brightness = s->brightness, + .contrast = s->contrast, + .saturation = s->saturation, + .hue = s->hue, + .gamma = s->gamma, + }; + + s->peak_detect_params = *pl_peak_detect_params( + .smoothing_period = s->smoothing, + .minimum_peak = s->min_peak, + .scene_threshold_low = s->scene_low, + .scene_threshold_high = s->scene_high, + .overshoot_margin = s->overshoot, + ); + + s->color_map_params = *pl_color_map_params( + .intent = s->intent, + .gamut_mode = gamut_mode, + .tone_mapping_function = tonemapping_funcs[s->tonemapping], + .tone_mapping_param = s->tonemapping_param, + .tone_mapping_mode = tonemapping_mode, + .inverse_tone_mapping = s->inverse_tonemapping, + .tone_mapping_crosstalk = s->crosstalk, + .lut_size = s->tonemapping_lut_size, + ); + + s->dither_params = *pl_dither_params( + .method = s->dithering, + .lut_size = s->dither_lut_size, + .temporal = s->dither_temporal, + ); + + s->cone_params = *pl_cone_params( + .cones = s->cones, + .strength = s->cone_str, + ); + + s->params = *pl_render_params( + .lut_entries = s->lut_entries, + .antiringing_strength = s->antiringing, + .background_transparency = 1.0f - (float) color_rgba[3] / UINT8_MAX, + .background_color = { + (float) color_rgba[0] / UINT8_MAX, + (float) color_rgba[1] / UINT8_MAX, + (float) color_rgba[2] / UINT8_MAX, + }, + + .deband_params = s->deband ? &s->deband_params : NULL, + .sigmoid_params = s->sigmoid ? &pl_sigmoid_default_params : NULL, + .color_adjustment = &s->color_adjustment, + .peak_detect_params = s->peakdetect ? &s->peak_detect_params : NULL, + .color_map_params = &s->color_map_params, + .dither_params = s->dithering >= 0 ? &s->dither_params : NULL, + .cone_params = s->cones ? &s->cone_params : NULL, + + .hooks = s->hooks, + .num_hooks = s->num_hooks, + + .skip_anti_aliasing = s->skip_aa, + .polar_cutoff = s->polar_cutoff, + .disable_linear_scaling = s->disable_linear, + .disable_builtin_scalers = s->disable_builtin, + .force_dither = s->force_dither, + .disable_fbos = s->disable_fbos, + ); + + RET(find_scaler(ctx, &s->params.upscaler, s->upscaler)); + RET(find_scaler(ctx, &s->params.downscaler, s->downscaler)); return 0; fail: @@ -327,6 +428,7 @@ static int libplacebo_init(AVFilterContext *avctx) s->out_format = AV_PIX_FMT_NONE; } + RET(update_settings(avctx)); RET(av_expr_parse(&s->crop_x_pexpr, s->crop_x_expr, var_names, NULL, NULL, NULL, NULL, 0, s)); RET(av_expr_parse(&s->crop_y_pexpr, s->crop_y_expr, var_names, @@ -449,14 +551,24 @@ static void libplacebo_uninit(AVFilterContext *avctx) av_expr_free(s->pos_h_pexpr); } +static int libplacebo_process_command(AVFilterContext *ctx, const char *cmd, + const char *arg, char *res, int res_len, + int flags) +{ + int err = 0; + RET(ff_filter_process_command(ctx, cmd, arg, res, res_len, flags)); + RET(update_settings(ctx)); + return 0; + +fail: + return err; +} + static int process_frames(AVFilterContext *avctx, AVFrame *out, AVFrame *in) { int err = 0, ok; LibplaceboContext *s = avctx->priv; - struct pl_render_params params; - enum pl_tone_map_mode tonemapping_mode = s->tonemapping_mode; const AVPixFmtDescriptor *outdesc = av_pix_fmt_desc_get(out->format); - enum pl_gamut_mode gamut_mode = s->gamut_mode; struct pl_frame image, target; ok = pl_map_avframe_ex(s->gpu, &image, pl_avframe_params( .frame = in, @@ -505,95 +617,7 @@ static int process_frames(AVFilterContext *avctx, AVFrame *out, AVFrame *in) pl_rect2df_aspect_set(&target.crop, aspect, s->pad_crop_ratio); } -#if FF_API_LIBPLACEBO_OPTS - /* backwards compatibility with older API */ - if (!tonemapping_mode && (s->desat_str >= 0.0f || s->desat_exp >= 0.0f)) { - float str = s->desat_str < 0.0f ? 0.9f : s->desat_str; - float exp = s->desat_exp < 0.0f ? 0.2f : s->desat_exp; - if (str >= 0.9f && exp <= 0.1f) { - tonemapping_mode = PL_TONE_MAP_RGB; - } else if (str > 0.1f) { - tonemapping_mode = PL_TONE_MAP_HYBRID; - } else { - tonemapping_mode = PL_TONE_MAP_LUMA; - } - } - - if (s->gamut_warning) - gamut_mode = PL_GAMUT_WARN; - if (s->gamut_clipping) - gamut_mode = PL_GAMUT_DESATURATE; -#endif - - /* Update render params */ - params = (struct pl_render_params) { - PL_RENDER_DEFAULTS - .lut_entries = s->lut_entries, - .antiringing_strength = s->antiringing, - - .deband_params = !s->deband ? NULL : pl_deband_params( - .iterations = s->deband_iterations, - .threshold = s->deband_threshold, - .radius = s->deband_radius, - .grain = s->deband_grain, - ), - - .sigmoid_params = s->sigmoid ? &pl_sigmoid_default_params : NULL, - - .color_adjustment = &(struct pl_color_adjustment) { - .brightness = s->brightness, - .contrast = s->contrast, - .saturation = s->saturation, - .hue = s->hue, - .gamma = s->gamma, - }, - - .peak_detect_params = !s->peakdetect ? NULL : pl_peak_detect_params( - .smoothing_period = s->smoothing, - .minimum_peak = s->min_peak, - .scene_threshold_low = s->scene_low, - .scene_threshold_high = s->scene_high, - .overshoot_margin = s->overshoot, - ), - - .color_map_params = pl_color_map_params( - .intent = s->intent, - .gamut_mode = gamut_mode, - .tone_mapping_function = tonemapping_funcs[s->tonemapping], - .tone_mapping_param = s->tonemapping_param, - .tone_mapping_mode = tonemapping_mode, - .inverse_tone_mapping = s->inverse_tonemapping, - .tone_mapping_crosstalk = s->crosstalk, - .lut_size = s->tonemapping_lut_size, - ), - - .dither_params = s->dithering < 0 ? NULL : pl_dither_params( - .method = s->dithering, - .lut_size = s->dither_lut_size, - .temporal = s->dither_temporal, - ), - - .cone_params = !s->cones ? NULL : pl_cone_params( - .cones = s->cones, - .strength = s->cone_str, - ), - - .hooks = s->hooks, - .num_hooks = s->num_hooks, - - .skip_anti_aliasing = s->skip_aa, - .polar_cutoff = s->polar_cutoff, - .disable_linear_scaling = s->disable_linear, - .disable_builtin_scalers = s->disable_builtin, - .force_dither = s->force_dither, - .disable_fbos = s->disable_fbos, - }; - - RET(find_scaler(avctx, ¶ms.upscaler, s->upscaler)); - RET(find_scaler(avctx, ¶ms.downscaler, s->downscaler)); - RET(parse_fillcolor(avctx, ¶ms, s->fillcolor)); - - pl_render_image(s->renderer, &image, &target, ¶ms); + pl_render_image(s->renderer, &image, &target, &s->params); pl_unmap_avframe(s->gpu, &image); if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { @@ -1049,7 +1073,7 @@ const AVFilter ff_vf_libplacebo = { .priv_size = sizeof(LibplaceboContext), .init = &libplacebo_init, .uninit = &libplacebo_uninit, - .process_command = &ff_filter_process_command, + .process_command = &libplacebo_process_command, FILTER_INPUTS(libplacebo_inputs), FILTER_OUTPUTS(libplacebo_outputs), FILTER_QUERY_FUNC(libplacebo_query_format), From patchwork Wed May 10 13:55:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41568 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867692pzb; Wed, 10 May 2023 06:56:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4KSAoTNYK83Xq80dIOw+we2Sua/fueOwaCF2hwrhEJkCglWdFJIJ83aLl9QiFpOZj6eT6Z X-Received: by 2002:a05:6402:b1a:b0:506:b2c7:2552 with SMTP id bm26-20020a0564020b1a00b00506b2c72552mr14765277edb.13.1683726964828; Wed, 10 May 2023 06:56:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726964; cv=none; d=google.com; s=arc-20160816; b=rUmRh1mHXZf/NNb5Vh/E+skly182OR1K1+mjdAzCLuDkdV4iglHnM7iaM8KaKrsKPK VePKu6MAN/Md0V7q7muilyBjHbgXdcJ06FfZJKKgPHz9mkEzfJPyEnKtmK4VW6oZu2Up 1wztuL6It0sM0sRE2aX1gefKZLcQxS1cc9WbSh2bEUnMCmZwHembQw0wtTsM12Yhhsk3 c7M8BQ+sV563a6qhpqxIOHvw/bVRyTfi0muP9MyAUDGLhFTys4ztfOhyRIzJ7bR9gP+C hcdIUEVA4bxSBe083RPFVbT0JqcttB5zAetHQvxumsrRHNZ/B27A2Mm2SY0RbyuYo2go OefQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=ONj4gi2KqIYeFRXLEWT6BmFSlvu7fsIdayCUcecY5MY=; b=nIWKTX1cXPqx8WPrzwWIIkd31GPQo9YzAkwebFaHu069zHgyQ4J0XlSQnhn4u0av+v qnX51ics+fpinhK6tigFsZG1fF4xB2qihrHMIvQmrqyW6rXA2FzZ4UPToL8Uw3XBA8rw /oFhF5DziU3DKeRkmvvxlM4anplgn6TN3Jg5121EiSijYFaCzTCLhlQOB078mM7rNcoA 8gqubsf5TrCsAcJuX7Wgr44d2Zu64almBLts0ETB9y+M5uDF+aa45U5H6GyEKKZRthHr hlp+iQybRrxIbjFHIEb4QLh1KhZ9subA0XFISs2dmvQvw+VQf+qWGyvtZ8mvQKNfEs7o WhZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=uy2GfcOM; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id y10-20020aa7d50a000000b0050bc4dd2ccbsi3236589edq.168.2023.05.10.06.56.04; Wed, 10 May 2023 06:56:04 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=uy2GfcOM; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 94E1768C173; Wed, 10 May 2023 16:55:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2227468B076 for ; Wed, 10 May 2023 16:55:17 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id 0323F42B4A; Wed, 10 May 2023 15:55:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726916; bh=bUN2g0OsJBPIegBokt4wtZy4QYgvMEpGohBuUfIjU1M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uy2GfcOMFaVGJfDwypvkIFZ5vC4e4d8GbQnn0M0fTuCu5u/F1fbYqxB6lrj0GJVxh exqfsS3cYBvxTdvWF9NKxZHQ5AnB0v6j2+umWPiRzprVOXgDsUX4H6mypR1nCUhIFd 9b4ryhNOxxA2h84mKxJpCGNUZ/bKJ+U0gsvAGxoQ= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:05 +0200 Message-Id: <20230510135509.4074-2-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230510135509.4074-1-ffmpeg@haasn.xyz> References: <20230510135509.4074-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/6] lavfi/vf_libplacebo: split and refactor logic X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: L1UrUfwZKXzv From: Niklas Haas This commit contains no functional change. The goal is merely to separate the highly intertwined `filter_frame` and `process_frames` functions into their separate concerns, specifically to separate frame uploading (which is now done directly in `filter_frame`) from emitting a frame (which is now done by a dedicated function `output_frame`). The overall idea here is to be able to ultimately call `output_frame` multiple times, to e.g. emit several output frames for a single input frame. --- libavfilter/vf_libplacebo.c | 172 +++++++++++++++++++----------------- 1 file changed, 93 insertions(+), 79 deletions(-) diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index 2ee9055c9d..e84f81c143 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -28,6 +28,14 @@ #include #include +/* Backwards compatibility with older libplacebo */ +#if PL_API_VER < 276 +static inline AVFrame *pl_get_mapped_avframe(const struct pl_frame *frame) +{ + return frame->user_data; +} +#endif + enum { TONE_MAP_AUTO, TONE_MAP_CLIP, @@ -564,35 +572,27 @@ fail: return err; } -static int process_frames(AVFilterContext *avctx, AVFrame *out, AVFrame *in) +static void update_crops(AVFilterContext *ctx, + struct pl_frame *image, + struct pl_frame *target, + const double target_pts) { - int err = 0, ok; - LibplaceboContext *s = avctx->priv; - const AVPixFmtDescriptor *outdesc = av_pix_fmt_desc_get(out->format); - struct pl_frame image, target; - ok = pl_map_avframe_ex(s->gpu, &image, pl_avframe_params( - .frame = in, - .tex = s->tex, - .map_dovi = s->apply_dovi, - )); - - if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { - ok &= pl_map_avframe_ex(s->gpu, &target, pl_avframe_params( - .frame = out, - .map_dovi = false, - )); - } else { - ok &= pl_frame_recreate_from_avframe(s->gpu, &target, s->tex + 4, out); - } + LibplaceboContext *s = ctx->priv; + const AVFrame *in = pl_get_mapped_avframe(image); + const AVFilterLink *inlink = ctx->inputs[0]; + const double in_pts = in->pts * av_q2d(inlink->time_base); - if (!ok) { - err = AVERROR_EXTERNAL; - goto fail; - } + s->var_values[VAR_IN_T] = s->var_values[VAR_T] = in_pts; + s->var_values[VAR_OUT_T] = s->var_values[VAR_OT] = target_pts; + s->var_values[VAR_N] = inlink->frame_count_out; - if (!s->apply_filmgrain) - image.film_grain.type = PL_FILM_GRAIN_NONE; + /* Clear these explicitly to avoid leaking previous frames' state */ + s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = NAN; + s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = NAN; + s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = NAN; + s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = NAN; + /* Evaluate crop/pos dimensions first, and placement second */ s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = av_expr_eval(s->crop_w_pexpr, s->var_values, NULL); s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = @@ -602,72 +602,39 @@ static int process_frames(AVFilterContext *avctx, AVFrame *out, AVFrame *in) s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = av_expr_eval(s->pos_h_pexpr, s->var_values, NULL); - image.crop.x0 = av_expr_eval(s->crop_x_pexpr, s->var_values, NULL); - image.crop.y0 = av_expr_eval(s->crop_y_pexpr, s->var_values, NULL); - image.crop.x1 = image.crop.x0 + s->var_values[VAR_CROP_W]; - image.crop.y1 = image.crop.y0 + s->var_values[VAR_CROP_H]; + image->crop.x0 = av_expr_eval(s->crop_x_pexpr, s->var_values, NULL); + image->crop.y0 = av_expr_eval(s->crop_y_pexpr, s->var_values, NULL); + image->crop.x1 = image->crop.x0 + s->var_values[VAR_CROP_W]; + image->crop.y1 = image->crop.y0 + s->var_values[VAR_CROP_H]; - target.crop.x0 = av_expr_eval(s->pos_x_pexpr, s->var_values, NULL); - target.crop.y0 = av_expr_eval(s->pos_y_pexpr, s->var_values, NULL); - target.crop.x1 = target.crop.x0 + s->var_values[VAR_POS_W]; - target.crop.y1 = target.crop.y0 + s->var_values[VAR_POS_H]; + target->crop.x0 = av_expr_eval(s->pos_x_pexpr, s->var_values, NULL); + target->crop.y0 = av_expr_eval(s->pos_y_pexpr, s->var_values, NULL); + target->crop.x1 = target->crop.x0 + s->var_values[VAR_POS_W]; + target->crop.y1 = target->crop.y0 + s->var_values[VAR_POS_H]; if (s->target_sar.num) { - float aspect = pl_rect2df_aspect(&target.crop) * av_q2d(s->target_sar); - pl_rect2df_aspect_set(&target.crop, aspect, s->pad_crop_ratio); - } - - pl_render_image(s->renderer, &image, &target, &s->params); - pl_unmap_avframe(s->gpu, &image); - - if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { - pl_unmap_avframe(s->gpu, &target); - } else if (!pl_download_avframe(s->gpu, &target, out)) { - err = AVERROR_EXTERNAL; - goto fail; + float aspect = pl_rect2df_aspect(&target->crop) * av_q2d(s->target_sar); + pl_rect2df_aspect_set(&target->crop, aspect, s->pad_crop_ratio); } - - /* Flush the command queues for performance */ - pl_gpu_flush(s->gpu); - return 0; - -fail: - pl_unmap_avframe(s->gpu, &image); - pl_unmap_avframe(s->gpu, &target); - return err; } -static int filter_frame(AVFilterLink *link, AVFrame *in) +/* Construct and emit an output frame for `image` */ +static int output_frame(AVFilterContext *ctx, struct pl_frame *image) { - int err, changed_csp; - AVFilterContext *ctx = link->dst; + int err = 0, ok, changed_csp; LibplaceboContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; - + const AVPixFmtDescriptor *outdesc = av_pix_fmt_desc_get(outlink->format); AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); - if (!out) { - err = AVERROR(ENOMEM); - goto fail; - } - - pl_log_level_update(s->log, get_log_level()); + const AVFrame *in = pl_get_mapped_avframe(image); + struct pl_frame target; + if (!out) + return AVERROR(ENOMEM); RET(av_frame_copy_props(out, in)); out->width = outlink->w; out->height = outlink->h; - /* Dynamic variables */ - s->var_values[VAR_IN_T] = s->var_values[VAR_T] = - in->pts == AV_NOPTS_VALUE ? NAN : in->pts * av_q2d(link->time_base); - s->var_values[VAR_OUT_T] = s->var_values[VAR_OT] = - out->pts == AV_NOPTS_VALUE ? NAN : out->pts * av_q2d(outlink->time_base); - s->var_values[VAR_N] = link->frame_count_out; - /* Will be evaluated/set by `process_frames` */ - s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = NAN; - s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = NAN; - s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = NAN; - s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = NAN; - if (s->apply_dovi && av_frame_get_side_data(in, AV_FRAME_DATA_DOVI_METADATA)) { /* Output of dovi reshaping is always BT.2020+PQ, so infer the correct * output colorspace defaults */ @@ -703,18 +670,65 @@ static int filter_frame(AVFilterLink *link, AVFrame *in) if (s->apply_filmgrain) av_frame_remove_side_data(out, AV_FRAME_DATA_FILM_GRAIN_PARAMS); - RET(process_frames(ctx, out, in)); + /* Map, render and unmap output frame */ + if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { + ok = pl_map_avframe_ex(s->gpu, &target, pl_avframe_params( + .frame = out, + .map_dovi = false, + )); + } else { + ok = pl_frame_recreate_from_avframe(s->gpu, &target, s->tex + 4, out); + } + if (!ok) { + err = AVERROR_EXTERNAL; + goto fail; + } - av_frame_free(&in); + update_crops(ctx, image, &target, out->pts * av_q2d(outlink->time_base)); + pl_render_image(s->renderer, image, &target, &s->params); + if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { + pl_unmap_avframe(s->gpu, &target); + } else if (!pl_download_avframe(s->gpu, &target, out)) { + err = AVERROR_EXTERNAL; + goto fail; + } return ff_filter_frame(outlink, out); fail: - av_frame_free(&in); av_frame_free(&out); return err; } +static int filter_frame(AVFilterLink *link, AVFrame *in) +{ + int ret, ok; + AVFilterContext *ctx = link->dst; + LibplaceboContext *s = ctx->priv; + struct pl_frame image; + + pl_log_level_update(s->log, get_log_level()); + + /* Map input frame */ + ok = pl_map_avframe_ex(s->gpu, &image, pl_avframe_params( + .frame = in, + .tex = s->tex, + .map_dovi = s->apply_dovi, + )); + av_frame_free(&in); + + if (!s->apply_filmgrain) + image.film_grain.type = PL_FILM_GRAIN_NONE; + + if (!ok) + return AVERROR_EXTERNAL; + + ret = output_frame(ctx, &image); + + pl_unmap_avframe(s->gpu, &image); + return ret; +} + static int libplacebo_query_format(AVFilterContext *ctx) { int err; From patchwork Wed May 10 13:55:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41565 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867337pzb; Wed, 10 May 2023 06:55:30 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5HUCQxTA49beOokquBUgaNnEgHKhpeDcUBz6qaGFxihT7kgNTxCkadQ+mChAhjdiUvSf7k X-Received: by 2002:a17:907:2d12:b0:969:98eb:3fdd with SMTP id gs18-20020a1709072d1200b0096998eb3fddmr8592445ejc.3.1683726930208; Wed, 10 May 2023 06:55:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726930; cv=none; d=google.com; s=arc-20160816; b=szusP4TNrN9/OQv9acqKE8uM+LUdXmCFt1mT72L5MCFLYur0uOswNVZVKe34UDiFnd 5gsyvvUot/8ID7Wxa9y3ZfOmh6Cg+DPdWC8G4359CokXd0il45KKb75QdNXoXx41O4k4 T8CTgwzb15KSMol6AHfiGEhD/JXDi9qt2B+1+DschKvspclXfdF+jdpHrPrp7+7Nax82 q3gfynpPi2vwNWuyaNrEIwm36QotuVfn1VmhdnveEiWVr09oqRWSaCj3GkTFuaSLF90k eZFG5XDg53lYqH+52Uh+zrRsiJ/PhRZP6ywgQ743rx2q9z1qekr4wk/fTp6TBwWTyJv7 ifFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=lg1eI3rqUfZcb22zJeBwUv2pElA4St6YMI/z4PcKkGo=; b=0d0jVGvPDNzBH6rL/Wxnl+111aIQdOlFUxCgDKiVQ9SNwBp0wJINnh4wWYPS2xxOkn gG0YvqipuRqGlH+76oso+IuD4qZsz67M2OVbQLhPhxRpFKFiJd5ZApXGFntu9x1j1+0i K1rPUKF8vJLEN76jtloyp1v4mLGv6p4alcgT4gQcHddfhemxbNkYrcrMAapYra+FKi9E +QO1hnhiZjWhHFV53Jq288KUXPN4+B7+HYAtjN/Dn44TNyQCK44tjrHMnPvNEg1DF2/h YW0NFbShjQbFE0FpeLmtZ4dH3Rtf3CaLOfb3yaWJLAYMI76QUMvv3P3l4QDOWfab1C3E g3Rw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=jqyePTkV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id m14-20020a17090607ce00b00958f6f65f2csi4066643ejc.413.2023.05.10.06.55.29; Wed, 10 May 2023 06:55:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=jqyePTkV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 41F4468C0C4; Wed, 10 May 2023 16:55:24 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1C665680BEB for ; Wed, 10 May 2023 16:55:17 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id 8D35D4994A; Wed, 10 May 2023 15:55:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726916; bh=s4wbMZ7MbpanTczp2cz7snSv8fvSZlB5HMwrOzuraO4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jqyePTkVLcrmnfwPs1bKBj49USSPlQJHfPPbi7X+jnPgrq380pa6SjbCtG5D+uGIJ urECctyl9akl61CdgE/k8sFyWHnO8qPDIzopV+NKiEQdLdqjt8p28RewFsa9b7Ne0F oCGBxRFEQdKDJsPZwS7WRvOp/niHfT5OHm3LMfP4= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:06 +0200 Message-Id: <20230510135509.4074-3-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230510135509.4074-1-ffmpeg@haasn.xyz> References: <20230510135509.4074-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/6] lavfi/vf_libplacebo: switch to pl_queue-based design X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: uJRy47UcKUFo From: Niklas Haas This does not leverage any immediate benefits, but refactors and prepares the codebase for upcoming changes, which will include the ability to do deinterlacing and resampling (frame mixing). --- libavfilter/vf_libplacebo.c | 214 ++++++++++++++++++++++++------------ 1 file changed, 144 insertions(+), 70 deletions(-) diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index e84f81c143..afb4e6a914 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -16,6 +16,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavutil/avassert.h" #include "libavutil/eval.h" #include "libavutil/file.h" #include "libavutil/opt.h" @@ -26,6 +27,7 @@ #include #include +#include #include /* Backwards compatibility with older libplacebo */ @@ -122,7 +124,8 @@ typedef struct LibplaceboContext { pl_vulkan vulkan; pl_gpu gpu; pl_renderer renderer; - pl_tex tex[8]; + pl_queue queue; + pl_tex tex[4]; /* settings */ char *out_format_string; @@ -392,6 +395,7 @@ static int update_settings(AVFilterContext *ctx) .num_hooks = s->num_hooks, .skip_anti_aliasing = s->skip_aa, + .skip_caching_single_frame = true, .polar_cutoff = s->polar_cutoff, .disable_linear_scaling = s->disable_linear, .disable_builtin_scalers = s->disable_builtin, @@ -518,6 +522,7 @@ static int init_vulkan(AVFilterContext *avctx) /* Create the renderer */ s->gpu = s->vulkan->gpu; s->renderer = pl_renderer_create(s->log, s->gpu); + s->queue = pl_queue_create(s->gpu); /* Parse the user shaders, if requested */ if (s->shader_bin_len) @@ -544,6 +549,7 @@ static void libplacebo_uninit(AVFilterContext *avctx) for (int i = 0; i < s->num_hooks; i++) pl_mpv_user_shader_destroy(&s->hooks[i]); pl_renderer_destroy(&s->renderer); + pl_queue_destroy(&s->queue); pl_vulkan_destroy(&s->vulkan); pl_log_destroy(&s->log); ff_vk_uninit(&s->vkctx); @@ -573,69 +579,92 @@ fail: } static void update_crops(AVFilterContext *ctx, - struct pl_frame *image, - struct pl_frame *target, - const double target_pts) + struct pl_frame_mix *mix, struct pl_frame *target, + uint64_t ref_sig, double base_pts) { LibplaceboContext *s = ctx->priv; - const AVFrame *in = pl_get_mapped_avframe(image); - const AVFilterLink *inlink = ctx->inputs[0]; - const double in_pts = in->pts * av_q2d(inlink->time_base); - - s->var_values[VAR_IN_T] = s->var_values[VAR_T] = in_pts; - s->var_values[VAR_OUT_T] = s->var_values[VAR_OT] = target_pts; - s->var_values[VAR_N] = inlink->frame_count_out; - - /* Clear these explicitly to avoid leaking previous frames' state */ - s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = NAN; - s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = NAN; - s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = NAN; - s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = NAN; - - /* Evaluate crop/pos dimensions first, and placement second */ - s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = - av_expr_eval(s->crop_w_pexpr, s->var_values, NULL); - s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = - av_expr_eval(s->crop_h_pexpr, s->var_values, NULL); - s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = - av_expr_eval(s->pos_w_pexpr, s->var_values, NULL); - s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = - av_expr_eval(s->pos_h_pexpr, s->var_values, NULL); - - image->crop.x0 = av_expr_eval(s->crop_x_pexpr, s->var_values, NULL); - image->crop.y0 = av_expr_eval(s->crop_y_pexpr, s->var_values, NULL); - image->crop.x1 = image->crop.x0 + s->var_values[VAR_CROP_W]; - image->crop.y1 = image->crop.y0 + s->var_values[VAR_CROP_H]; - - target->crop.x0 = av_expr_eval(s->pos_x_pexpr, s->var_values, NULL); - target->crop.y0 = av_expr_eval(s->pos_y_pexpr, s->var_values, NULL); - target->crop.x1 = target->crop.x0 + s->var_values[VAR_POS_W]; - target->crop.y1 = target->crop.y0 + s->var_values[VAR_POS_H]; - - if (s->target_sar.num) { - float aspect = pl_rect2df_aspect(&target->crop) * av_q2d(s->target_sar); - pl_rect2df_aspect_set(&target->crop, aspect, s->pad_crop_ratio); + + for (int i = 0; i < mix->num_frames; i++) { + // Mutate the `pl_frame.crop` fields in-place. This is fine because we + // own the entire pl_queue, and hence, the pointed-at frames. + struct pl_frame *image = (struct pl_frame *) mix->frames[i]; + double image_pts = base_pts + mix->timestamps[i]; + + /* Update dynamic variables */ + s->var_values[VAR_IN_T] = s->var_values[VAR_T] = image_pts; + s->var_values[VAR_OUT_T] = s->var_values[VAR_OT] = base_pts; + s->var_values[VAR_N] = ctx->outputs[0]->frame_count_out; + + /* Clear these explicitly to avoid leaking previous frames' state */ + s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = NAN; + s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = NAN; + s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = NAN; + s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = NAN; + + /* Compute dimensions first and placement second */ + s->var_values[VAR_CROP_W] = s->var_values[VAR_CW] = + av_expr_eval(s->crop_w_pexpr, s->var_values, NULL); + s->var_values[VAR_CROP_H] = s->var_values[VAR_CH] = + av_expr_eval(s->crop_h_pexpr, s->var_values, NULL); + s->var_values[VAR_POS_W] = s->var_values[VAR_PW] = + av_expr_eval(s->pos_w_pexpr, s->var_values, NULL); + s->var_values[VAR_POS_H] = s->var_values[VAR_PH] = + av_expr_eval(s->pos_h_pexpr, s->var_values, NULL); + + image->crop.x0 = av_expr_eval(s->crop_x_pexpr, s->var_values, NULL); + image->crop.y0 = av_expr_eval(s->crop_y_pexpr, s->var_values, NULL); + image->crop.x1 = image->crop.x0 + s->var_values[VAR_CROP_W]; + image->crop.y1 = image->crop.y0 + s->var_values[VAR_CROP_H]; + + if (mix->signatures[i] == ref_sig) { + /* Only update the target crop once, for the 'reference' frame */ + target->crop.x0 = av_expr_eval(s->pos_x_pexpr, s->var_values, NULL); + target->crop.y0 = av_expr_eval(s->pos_y_pexpr, s->var_values, NULL); + target->crop.x1 = target->crop.x0 + s->var_values[VAR_POS_W]; + target->crop.y1 = target->crop.y0 + s->var_values[VAR_POS_H]; + + if (s->target_sar.num) { + float aspect = pl_rect2df_aspect(&target->crop) * av_q2d(s->target_sar); + pl_rect2df_aspect_set(&target->crop, aspect, s->pad_crop_ratio); + } + } } } -/* Construct and emit an output frame for `image` */ -static int output_frame(AVFilterContext *ctx, struct pl_frame *image) +/* Construct and emit an output frame for a given frame mix */ +static int output_frame_mix(AVFilterContext *ctx, + struct pl_frame_mix *mix, + int64_t pts) { int err = 0, ok, changed_csp; LibplaceboContext *s = ctx->priv; AVFilterLink *outlink = ctx->outputs[0]; const AVPixFmtDescriptor *outdesc = av_pix_fmt_desc_get(outlink->format); - AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); - const AVFrame *in = pl_get_mapped_avframe(image); struct pl_frame target; + const AVFrame *ref; + AVFrame *out; + uint64_t ref_sig; + if (!mix->num_frames) + return 0; + + out = ff_get_video_buffer(outlink, outlink->w, outlink->h); if (!out) return AVERROR(ENOMEM); - RET(av_frame_copy_props(out, in)); + /* Use the last frame before current PTS value as reference */ + for (int i = 0; i < mix->num_frames; i++) { + if (i && mix->timestamps[i] > 0.0f) + break; + ref = pl_get_mapped_avframe(mix->frames[i]); + ref_sig = mix->signatures[i]; + } + + RET(av_frame_copy_props(out, ref)); + out->pts = pts; out->width = outlink->w; out->height = outlink->h; - if (s->apply_dovi && av_frame_get_side_data(in, AV_FRAME_DATA_DOVI_METADATA)) { + if (s->apply_dovi && av_frame_get_side_data(ref, AV_FRAME_DATA_DOVI_METADATA)) { /* Output of dovi reshaping is always BT.2020+PQ, so infer the correct * output colorspace defaults */ out->colorspace = AVCOL_SPC_BT2020_NCL; @@ -652,10 +681,10 @@ static int output_frame(AVFilterContext *ctx, struct pl_frame *image) if (s->color_primaries >= 0) out->color_primaries = s->color_primaries; - changed_csp = in->colorspace != out->colorspace || - in->color_range != out->color_range || - in->color_trc != out->color_trc || - in->color_primaries != out->color_primaries; + changed_csp = ref->colorspace != out->colorspace || + ref->color_range != out->color_range || + ref->color_trc != out->color_trc || + ref->color_primaries != out->color_primaries; /* Strip side data if no longer relevant */ if (changed_csp) { @@ -677,15 +706,15 @@ static int output_frame(AVFilterContext *ctx, struct pl_frame *image) .map_dovi = false, )); } else { - ok = pl_frame_recreate_from_avframe(s->gpu, &target, s->tex + 4, out); + ok = pl_frame_recreate_from_avframe(s->gpu, &target, s->tex, out); } if (!ok) { err = AVERROR_EXTERNAL; goto fail; } - update_crops(ctx, image, &target, out->pts * av_q2d(outlink->time_base)); - pl_render_image(s->renderer, image, &target, &s->params); + update_crops(ctx, mix, &target, ref_sig, out->pts * av_q2d(outlink->time_base)); + pl_render_image_mix(s->renderer, mix, &target, &s->params); if (outdesc->flags & AV_PIX_FMT_FLAG_HWACCEL) { pl_unmap_avframe(s->gpu, &target); @@ -700,33 +729,78 @@ fail: return err; } +static bool map_frame(pl_gpu gpu, pl_tex *tex, + const struct pl_source_frame *src, + struct pl_frame *out) +{ + AVFrame *avframe = src->frame_data; + LibplaceboContext *s = avframe->opaque; + bool ok = pl_map_avframe_ex(gpu, out, pl_avframe_params( + .frame = avframe, + .tex = tex, + .map_dovi = s->apply_dovi, + )); + + if (!s->apply_filmgrain) + out->film_grain.type = PL_FILM_GRAIN_NONE; + + av_frame_free(&avframe); + return ok; +} + +static void unmap_frame(pl_gpu gpu, struct pl_frame *frame, + const struct pl_source_frame *src) +{ + pl_unmap_avframe(gpu, frame); +} + +static void discard_frame(const struct pl_source_frame *src) +{ + AVFrame *avframe = src->frame_data; + av_frame_free(&avframe); +} + static int filter_frame(AVFilterLink *link, AVFrame *in) { - int ret, ok; AVFilterContext *ctx = link->dst; LibplaceboContext *s = ctx->priv; - struct pl_frame image; + AVFilterLink *outlink = ctx->outputs[0]; + enum pl_queue_status status; + struct pl_frame_mix mix; pl_log_level_update(s->log, get_log_level()); - /* Map input frame */ - ok = pl_map_avframe_ex(s->gpu, &image, pl_avframe_params( - .frame = in, - .tex = s->tex, - .map_dovi = s->apply_dovi, + /* Push input frame */ + in->opaque = s; + pl_queue_push(s->queue, &(struct pl_source_frame) { + .pts = in->pts * av_q2d(link->time_base), + .duration = in->duration * av_q2d(link->time_base), + .first_field = pl_field_from_avframe(in), + .frame_data = in, + .map = map_frame, + .unmap = unmap_frame, + .discard = discard_frame, + }); + + /* Immediately return an output frame for the same PTS */ + av_assert1(!av_cmp_q(link->time_base, outlink->time_base)); + status = pl_queue_update(s->queue, &mix, pl_queue_params( + .pts = in->pts * av_q2d(outlink->time_base), + .radius = pl_frame_mix_radius(&s->params), + .vsync_duration = av_q2d(av_inv_q(outlink->frame_rate)), )); - av_frame_free(&in); - - if (!s->apply_filmgrain) - image.film_grain.type = PL_FILM_GRAIN_NONE; - if (!ok) + switch (status) { + case PL_QUEUE_MORE: // TODO: switch to activate() and handle properly + case PL_QUEUE_OK: + return output_frame_mix(ctx, &mix, in->pts); + case PL_QUEUE_EOF: + return 0; + case PL_QUEUE_ERR: return AVERROR_EXTERNAL; + } - ret = output_frame(ctx, &image); - - pl_unmap_avframe(s->gpu, &image); - return ret; + return AVERROR_BUG; } static int libplacebo_query_format(AVFilterContext *ctx) From patchwork Wed May 10 13:55:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41567 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867644pzb; Wed, 10 May 2023 06:56:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ65hHLAJ5C7g61RLWzkIi24Nt8KHocmyUXwpoFSkzE5VVMJCVW+v2d52dKg3DuzwQToZAhm X-Received: by 2002:a17:907:3184:b0:94a:5911:b475 with SMTP id xe4-20020a170907318400b0094a5911b475mr16283063ejb.52.1683726960496; Wed, 10 May 2023 06:56:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726960; cv=none; d=google.com; s=arc-20160816; b=jmW2yfuXI3ljuT5eG8dDZGgVPBlovOvAe5U/BwXc+06pMCnX1/OZHsLGSkuWrRCMoS 6aIzAKQSr8TkODZkG+3r6p207OfAkc4bAP7bvTZekxDE4+UXHOyuMV5X29LDBc7AF1hD 0PdSfJi8Uq4b7mgUQc+rRO9mZaInWRUzcYibkibD2ZuoLgKk6RyVmazKtsbWrT69CnvD hkTh4Px67e9yUTZp0o5AafloZYMKPhDia+U7mLZ4S7qYOjyUHBQ6h9O9W0Hn6CRR4gfk He+hNiyD/i1cKbIn+j9a4LHgtd5J5DS03TB5RQxYhdK0zKA86MBykTDQ7/DsjAQ5TBEy La2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=9506sgGfKKix1kg4jtBF30o+glDF3qXUwCiVnl/kIjs=; b=tcOGbgaLi6JcFZAbvT3b3+nTld2p0oM7WixMi+HKB8SP1oXpuoe3VPYp2KQAKunXNS ONLWwE9y+Kh/6fYSVAFSCFElCkDjkrfSnSsFUiWCNDwcydqo+ImKEJNiFmu73VHhcTqg mTZP3iyKfALFzL6kuV2DDBU1+kpdieJVybLLJKynQfouEFfpTt0JVt5gCw+rrBXn18RM xZyi4iOWymowV1qavFID4QZ8mXmM6gnljtyXL/CmJZ7r+9A+4SPjqE2iOv9zzDbc3agi 6dqasid1p0e7uWvzO+zc7HkdpFErFqIpgWAovDiZR22lTGPJxINSHh5ecwROD5kUdhcw jlrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=tY05xqAi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a21-20020a1709062b1500b00965a6d6b536si3320384ejg.335.2023.05.10.06.55.53; Wed, 10 May 2023 06:56:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=tY05xqAi; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 762DB68C164; Wed, 10 May 2023 16:55:26 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 7CD0E68BBE3 for ; Wed, 10 May 2023 16:55:17 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id 2D4A349A4F; Wed, 10 May 2023 15:55:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726917; bh=NKf/5NW3O8iMBckLuaUD6ZJgRDdIiLgAcZeBvrk8Auc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tY05xqAioS9kLxri+gGNb9st34vcRAWQqHgIcsf76owjt+DbRo2zzdmL0Cccs3T9s TjdhTencaTS9IwVC6u31P6RDdb5fFzk7DD10FA5Qxyh93hly8E76N0oG30aPpUF+9N fHbH+TbWgkySYXWfvdAY7RAkDCxOp0YfkRaCxwKU= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:07 +0200 Message-Id: <20230510135509.4074-4-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230510135509.4074-1-ffmpeg@haasn.xyz> References: <20230510135509.4074-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 4/6] lavfi/vf_libplacebo: switch to activate() X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: pMpDpfRwZFce From: Niklas Haas To present compatibility with the current behavior, we keep track of a FIFO of exact frame timestamps that we want to output to the user. In practice, this is essentially equivalent to the current filter_frame() code, but this design allows us to scale to more complicated use cases in the future - for example, insertion of intermediate frames (deinterlacing, frame doubling, conversion to fixed fps, ...) --- libavfilter/vf_libplacebo.c | 113 +++++++++++++++++++++++++----------- 1 file changed, 80 insertions(+), 33 deletions(-) diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index afb4e6a914..d8128351c8 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -18,10 +18,12 @@ #include "libavutil/avassert.h" #include "libavutil/eval.h" +#include "libavutil/fifo.h" #include "libavutil/file.h" #include "libavutil/opt.h" #include "libavutil/parseutils.h" #include "internal.h" +#include "filters.h" #include "vulkan_filter.h" #include "scale_eval.h" @@ -127,6 +129,9 @@ typedef struct LibplaceboContext { pl_queue queue; pl_tex tex[4]; + /* filter state */ + AVFifo *out_pts; ///< timestamps of wanted output frames + /* settings */ char *out_format_string; enum AVPixelFormat out_format; @@ -458,6 +463,9 @@ static int libplacebo_init(AVFilterContext *avctx) RET(av_expr_parse(&s->pos_h_pexpr, s->pos_h_expr, var_names, NULL, NULL, NULL, NULL, 0, s)); + /* Initialize dynamic filter state */ + s->out_pts = av_fifo_alloc2(1, sizeof(int64_t), AV_FIFO_FLAG_AUTO_GROW); + /* Note: s->vulkan etc. are initialized later, when hwctx is available */ return 0; @@ -563,6 +571,7 @@ static void libplacebo_uninit(AVFilterContext *avctx) av_expr_free(s->pos_y_pexpr); av_expr_free(s->pos_w_pexpr); av_expr_free(s->pos_h_pexpr); + av_fifo_freep2(&s->out_pts); } static int libplacebo_process_command(AVFilterContext *ctx, const char *cmd, @@ -760,47 +769,85 @@ static void discard_frame(const struct pl_source_frame *src) av_frame_free(&avframe); } -static int filter_frame(AVFilterLink *link, AVFrame *in) +static int libplacebo_activate(AVFilterContext *ctx) { - AVFilterContext *ctx = link->dst; + int ret, status; LibplaceboContext *s = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; AVFilterLink *outlink = ctx->outputs[0]; - enum pl_queue_status status; - struct pl_frame_mix mix; + AVFrame *in; + int64_t pts; + FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink); pl_log_level_update(s->log, get_log_level()); - /* Push input frame */ - in->opaque = s; - pl_queue_push(s->queue, &(struct pl_source_frame) { - .pts = in->pts * av_q2d(link->time_base), - .duration = in->duration * av_q2d(link->time_base), - .first_field = pl_field_from_avframe(in), - .frame_data = in, - .map = map_frame, - .unmap = unmap_frame, - .discard = discard_frame, - }); - - /* Immediately return an output frame for the same PTS */ - av_assert1(!av_cmp_q(link->time_base, outlink->time_base)); - status = pl_queue_update(s->queue, &mix, pl_queue_params( - .pts = in->pts * av_q2d(outlink->time_base), - .radius = pl_frame_mix_radius(&s->params), - .vsync_duration = av_q2d(av_inv_q(outlink->frame_rate)), - )); + while ((ret = ff_inlink_consume_frame(inlink, &in)) > 0) { + in->opaque = s; + pl_queue_push(s->queue, &(struct pl_source_frame) { + .pts = in->pts * av_q2d(inlink->time_base), + .duration = in->duration * av_q2d(inlink->time_base), + .first_field = pl_field_from_avframe(in), + .frame_data = in, + .map = map_frame, + .unmap = unmap_frame, + .discard = discard_frame, + }); + + /* Internally queue an output frame for the same PTS */ + av_assert1(!av_cmp_q(link->time_base, outlink->time_base)); + av_fifo_write(s->out_pts, &in->pts, 1); + } - switch (status) { - case PL_QUEUE_MORE: // TODO: switch to activate() and handle properly - case PL_QUEUE_OK: - return output_frame_mix(ctx, &mix, in->pts); - case PL_QUEUE_EOF: - return 0; - case PL_QUEUE_ERR: - return AVERROR_EXTERNAL; + if (ret < 0) + return ret; + + if (ff_inlink_acknowledge_status(inlink, &status, &pts)) { + pts = av_rescale_q_rnd(pts, inlink->time_base, outlink->time_base, + AV_ROUND_UP); + if (status == AVERROR_EOF) { + /* Signal EOF to pl_queue, and enqueue this output frame to + * make sure we see PL_QUEUE_EOF returned eventually */ + pl_queue_push(s->queue, NULL); + av_fifo_write(s->out_pts, &pts, 1); + } else { + ff_outlink_set_status(outlink, status, pts); + return 0; + } + } + + if (ff_outlink_frame_wanted(outlink)) { + struct pl_frame_mix mix; + enum pl_queue_status ret; + + if (av_fifo_peek(s->out_pts, &pts, 1, 0) < 0) { + ff_inlink_request_frame(inlink); + return 0; + } + + ret = pl_queue_update(s->queue, &mix, pl_queue_params( + .pts = pts * av_q2d(outlink->time_base), + .radius = pl_frame_mix_radius(&s->params), + .vsync_duration = av_q2d(av_inv_q(outlink->frame_rate)), + )); + + switch (ret) { + case PL_QUEUE_MORE: + ff_inlink_request_frame(inlink); + return 0; + case PL_QUEUE_OK: + av_fifo_drain2(s->out_pts, 1); + return output_frame_mix(ctx, &mix, pts); + case PL_QUEUE_EOF: + ff_outlink_set_status(outlink, AVERROR_EOF, pts); + return 0; + case PL_QUEUE_ERR: + return AVERROR_EXTERNAL; + } + + return AVERROR_BUG; } - return AVERROR_BUG; + return FFERROR_NOT_READY; } static int libplacebo_query_format(AVFilterContext *ctx) @@ -1142,7 +1189,6 @@ static const AVFilterPad libplacebo_inputs[] = { { .name = "default", .type = AVMEDIA_TYPE_VIDEO, - .filter_frame = &filter_frame, .config_props = &libplacebo_config_input, }, }; @@ -1161,6 +1207,7 @@ const AVFilter ff_vf_libplacebo = { .priv_size = sizeof(LibplaceboContext), .init = &libplacebo_init, .uninit = &libplacebo_uninit, + .activate = &libplacebo_activate, .process_command = &libplacebo_process_command, FILTER_INPUTS(libplacebo_inputs), FILTER_OUTPUTS(libplacebo_outputs), From patchwork Wed May 10 13:55:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41569 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867779pzb; Wed, 10 May 2023 06:56:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4V87RtFWN3I05zOKuabEwwQlsBjgln7xKc8S9s+gR3++e37q/ssIPGt9XB+CsyPL3NrHvv X-Received: by 2002:a17:907:94c2:b0:966:3114:c790 with SMTP id dn2-20020a17090794c200b009663114c790mr13918533ejc.37.1683726975097; Wed, 10 May 2023 06:56:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726975; cv=none; d=google.com; s=arc-20160816; b=GZNwL7HVmSuON8skCSPYd0PVU5Wk5JsLXDz/Fv02u0fdXsdFMD8ggeXtjYVMPdn68o MIsQZDR3iuX8CGmXROVE/Uah+FdVEFddbYvVEtac2vQo8alGa0xVeGlQxczXVt8rlkv2 ejWfcnNbljd1FM3cY2qubZ3OyKe+j8nunA3zpE0TuY0+PQG1tbRGizCy9MAPcFCDdXwM SesTiFTLAN7Kn4KeBHkXZcrPdH35sVJhkchDaqRFt0cMMzmHxdlglFsUV7aWNf3tVOft tVcq0pGm7rcB7LVgwIaVhsmKAH+UOE3NqgD1mRJz1/JpKBeGIXaSgSiuwU+wHg8aTcT/ mqsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=2ahvfivKRDm/MlbcTI2D0mTCITAtudcAq5kLqdvY5mI=; b=O46AXNQD2jkjiTYdDfRHL0e1ytYppGn0UKg4qboQ9QzjppkCOHbv0MwGbJrgNnXMhy iaBqiDd5g7G6oOq785psQ6gxINgtK8RuUFrwNdAy1hz7g/FLXYuBzVUy3GrQ65kgJ46U VseijdahZSGT3BRv0WvAiSj0HiHmxNDCqjPSMFUDHEHqZl0TVOCSP+QCsGo0Ldb/EaOT O9G8ZkXxxhQYp3zt4gYhqZ/1E78iQc2r9zaVk5LbXJpaJBqYC2d1llIv/04GXI6WK4Ie VrstWNbQh1HL3BAEVbeBUf2mamRKT1NoHzgAhvG2TdY2HiTOJc9lF5kN9lh8ivdFrdpk LC7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=u66Qyz0a; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id qf30-20020a1709077f1e00b0094f76af1aa7si575265ejc.655.2023.05.10.06.56.14; Wed, 10 May 2023 06:56:15 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=u66Qyz0a; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id A3B9568C189; Wed, 10 May 2023 16:55:28 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0983668B076 for ; Wed, 10 May 2023 16:55:18 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id B8BE949F37; Wed, 10 May 2023 15:55:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726917; bh=9zLvwUWkkeB4P8bE0QMkD5XsMTYys7oetjCpNvwdkbA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u66Qyz0aWTTHSowgEvVBTPTUSoj4b6EL8bmTPe488yM/fWhTpoKjXUj8F5YjZTtNY uowsY+L5xCQ1wTAYyQL7tJRy+s2WVJ/9Qf8KI5DQlV7DeA5ITFrPt0yvXZU5UZnxh/ mOSBn/0ixgPBDykKjysEHbtBHD136SiknECqLHFI= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:08 +0200 Message-Id: <20230510135509.4074-5-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230510135509.4074-1-ffmpeg@haasn.xyz> References: <20230510135509.4074-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 5/6] lavfi/vf_libplacebo: allow fps conversion X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: T4tgxlSptvfA From: Niklas Haas This exposes libplacebo's frame mixing functionality to vf_libplacebo, by allowing users to specify a desired target fps to output at. Incoming frames will be smoothly resampled (in a manner determined by the `frame_mixer` option, to be added in the next commit). To generate a consistently timed output stream, we directly use the desired framerate as the timebase, and simply output frames in sequential order (tracked by the number of frames output so far). --- doc/filters.texi | 7 +++++++ libavfilter/vf_libplacebo.c | 38 ++++++++++++++++++++++++++++++------- 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 206fb23785..3e17ba1654 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -16034,6 +16034,13 @@ and @code{(oh-ph)/2}. Set the output placement width/height expressions, default values are @code{ow} and @code{oh}. +@item fps +Set the output frame rate. This can be rational, e.g. @code{60000/1001}. If +set to the special string @code{none} (the default), input timestamps will +instead be passed through to the output unmodified. Otherwise, the input video +frames will be interpolated as necessary to rescale the video to the specified +target framerate, in a manner as determined by the @option{frame_mixer} option. + @item format Set the output format override. If unset (the default), frames will be output in the same format as the respective input frames. Otherwise, format conversion diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index d8128351c8..2bec942358 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -139,6 +139,8 @@ typedef struct LibplaceboContext { double var_values[VAR_VARS_NB]; char *w_expr; char *h_expr; + char *fps_string; + AVRational fps; ///< parsed FPS, or 0/0 for "none" char *crop_x_expr, *crop_y_expr; char *crop_w_expr, *crop_h_expr; char *pos_x_expr, *pos_y_expr; @@ -166,6 +168,7 @@ typedef struct LibplaceboContext { float antiringing; int sigmoid; int skip_aa; + int skip_cache; float polar_cutoff; int disable_linear; int disable_builtin; @@ -400,7 +403,7 @@ static int update_settings(AVFilterContext *ctx) .num_hooks = s->num_hooks, .skip_anti_aliasing = s->skip_aa, - .skip_caching_single_frame = true, + .skip_caching_single_frame = s->skip_cache, .polar_cutoff = s->polar_cutoff, .disable_linear_scaling = s->disable_linear, .disable_builtin_scalers = s->disable_builtin, @@ -465,6 +468,8 @@ static int libplacebo_init(AVFilterContext *avctx) /* Initialize dynamic filter state */ s->out_pts = av_fifo_alloc2(1, sizeof(int64_t), AV_FIFO_FLAG_AUTO_GROW); + if (strcmp(s->fps_string, "none") != 0) + RET(av_parse_video_rate(&s->fps, s->fps_string)); /* Note: s->vulkan etc. are initialized later, when hwctx is available */ return 0; @@ -672,6 +677,8 @@ static int output_frame_mix(AVFilterContext *ctx, out->pts = pts; out->width = outlink->w; out->height = outlink->h; + if (s->fps.num) + out->duration = 1; if (s->apply_dovi && av_frame_get_side_data(ref, AV_FRAME_DATA_DOVI_METADATA)) { /* Output of dovi reshaping is always BT.2020+PQ, so infer the correct @@ -793,9 +800,11 @@ static int libplacebo_activate(AVFilterContext *ctx) .discard = discard_frame, }); - /* Internally queue an output frame for the same PTS */ - av_assert1(!av_cmp_q(link->time_base, outlink->time_base)); - av_fifo_write(s->out_pts, &in->pts, 1); + if (!s->fps.num) { + /* Internally queue an output frame for the same PTS */ + av_assert1(!av_cmp_q(link->time_base, outlink->time_base)); + av_fifo_write(s->out_pts, &in->pts, 1); + } } if (ret < 0) @@ -808,7 +817,8 @@ static int libplacebo_activate(AVFilterContext *ctx) /* Signal EOF to pl_queue, and enqueue this output frame to * make sure we see PL_QUEUE_EOF returned eventually */ pl_queue_push(s->queue, NULL); - av_fifo_write(s->out_pts, &pts, 1); + if (!s->fps.num) + av_fifo_write(s->out_pts, &pts, 1); } else { ff_outlink_set_status(outlink, status, pts); return 0; @@ -819,7 +829,9 @@ static int libplacebo_activate(AVFilterContext *ctx) struct pl_frame_mix mix; enum pl_queue_status ret; - if (av_fifo_peek(s->out_pts, &pts, 1, 0) < 0) { + if (s->fps.num) { + pts = outlink->frame_count_out; + } else if (av_fifo_peek(s->out_pts, &pts, 1, 0) < 0) { ff_inlink_request_frame(inlink); return 0; } @@ -835,7 +847,8 @@ static int libplacebo_activate(AVFilterContext *ctx) ff_inlink_request_frame(inlink); return 0; case PL_QUEUE_OK: - av_fifo_drain2(s->out_pts, 1); + if (!s->fps.num) + av_fifo_drain2(s->out_pts, 1); return output_frame_mix(ctx, &mix, pts); case PL_QUEUE_EOF: ff_outlink_set_status(outlink, AVERROR_EOF, pts); @@ -939,6 +952,7 @@ static int libplacebo_config_output(AVFilterLink *outlink) AVVulkanFramesContext *vkfc; AVRational scale_sar; + /* Frame dimensions */ RET(ff_scale_eval_dimensions(s, s->w_expr, s->h_expr, inlink, outlink, &outlink->w, &outlink->h)); @@ -962,6 +976,15 @@ static int libplacebo_config_output(AVFilterLink *outlink) outlink->sample_aspect_ratio = scale_sar; } + /* Frame rate */ + if (s->fps.num) { + outlink->frame_rate = s->fps; + outlink->time_base = av_inv_q(s->fps); + s->skip_cache = av_cmp_q(inlink->frame_rate, s->fps) > 0; + } else { + s->skip_cache = true; + } + /* Static variables */ s->var_values[VAR_IN_W] = s->var_values[VAR_IW] = inlink->w; s->var_values[VAR_IN_H] = s->var_values[VAR_IH] = inlink->h; @@ -1006,6 +1029,7 @@ fail: static const AVOption libplacebo_options[] = { { "w", "Output video frame width", OFFSET(w_expr), AV_OPT_TYPE_STRING, {.str = "iw"}, .flags = STATIC }, { "h", "Output video frame height", OFFSET(h_expr), AV_OPT_TYPE_STRING, {.str = "ih"}, .flags = STATIC }, + { "fps", "Output video frame rate", OFFSET(fps_string), AV_OPT_TYPE_STRING, {.str = "none"}, .flags = STATIC }, { "crop_x", "Input video crop x", OFFSET(crop_x_expr), AV_OPT_TYPE_STRING, {.str = "(iw-cw)/2"}, .flags = DYNAMIC }, { "crop_y", "Input video crop y", OFFSET(crop_y_expr), AV_OPT_TYPE_STRING, {.str = "(ih-ch)/2"}, .flags = DYNAMIC }, { "crop_w", "Input video crop w", OFFSET(crop_w_expr), AV_OPT_TYPE_STRING, {.str = "iw"}, .flags = DYNAMIC }, From patchwork Wed May 10 13:55:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niklas Haas X-Patchwork-Id: 41570 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp3867876pzb; Wed, 10 May 2023 06:56:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7ilWnLjnPlELrA83SzHwacWL77IeUPWDJlkeTdfl5uFpKxTtOHd5W+0EPxYouLPWcUQsoS X-Received: by 2002:a17:907:6da0:b0:96a:4409:ef96 with SMTP id sb32-20020a1709076da000b0096a4409ef96mr523511ejc.77.1683726984344; Wed, 10 May 2023 06:56:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683726984; cv=none; d=google.com; s=arc-20160816; b=e81WBdRL9bDN3mrbkCZkUEMy3NH+uC0vQCluOUubt8UbxMPEhcdVccsLu6MZGIG4l2 EZvhm5H+rVsGTLAiuxiGG8SUtNKI42YbAEbs5idRCD09geAxWHjYLQ2GX8vqzhM3XAGZ SI8fodShKfaO/8HRnu2HK1LkGp4pFzF/xkfaxzfCwJpJX+8i2tsfqBTUmcuWEf3FxJWJ 3D0ltdjt40caUwDwCqWADuXlDnPrUg1tVeodbdQv6QLQx9Lczwknl40fVJGyEvCzyrHV ov1/CZXN78VG7xN4BSJYX9YyZe0iiDLb/Nz6KiLfjdmRiNbqUTYSGGZhc6b4D4sCjY6V BHHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=307kUyDzXVmmLIkDGMoADRqSWbOp9xQUAoiUyOzLmgA=; b=ogsKW/1NpoPM+87/0yj4mKJ9Fgxg7SQqQIpyl/AXeL7rs8P5XxUHuJiOek2sp+NvPQ G5wW7Gp2zCSgrqzXhlETlzQVxx2Dt34x/wnBiqW0nN6V1/4zdsAb+/T6n/xLUxYu5lt2 2EO/DEcFkpgyePQb893NaTf5ogrJIlPIAPGWpGC5WubXxhzX4s3I3fiUCMbbfWfNDD77 A/vjMHQ+qICzTlmncBkV2MIeBpNdKQs4fScKoT4hn6a3iaZOyS1VUZ3DoGAivtaLJHA7 nvrkdlikR8DzwoR+wk9RmMBLK1ZrYllG1GtZB4NYEh05K0JNzhmNbZbHKZqwI+HQumhw v2qQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=HvX4F37R; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b22-20020a1709062b5600b0096651724240si3135322ejg.286.2023.05.10.06.56.24; Wed, 10 May 2023 06:56:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@haasn.xyz header.s=mail header.b=HvX4F37R; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BA62868C19B; Wed, 10 May 2023 16:55:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from haasn.dev (haasn.dev [78.46.187.166]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8B98968C170 for ; Wed, 10 May 2023 16:55:22 +0300 (EEST) Received: from localhost (217-74-0-168.hsi.r-kom.net [217.74.0.168]) by haasn.dev (Postfix) with ESMTPSA id 568DE4AA43; Wed, 10 May 2023 15:55:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=haasn.xyz; s=mail; t=1683726918; bh=KN4fsxd7kygArHsiO75slviY5jotxqXTUsdtoc+2nDY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HvX4F37RvE4wRuyemvVYnh9KZLNsbe6HwMOLkPJk4M8VqQ2adg2BlmnbEZQQiUAFi aqqGSLCzuWTEbbldwqRg7/BFtMM4TBNQ2GmfKpEluRVKCsrP87b91xchImsBUxPWhB cgA5MB4WJc+wHiKWs5pHeImTWkbBMAYZtBE+2YKo= From: Niklas Haas To: ffmpeg-devel@ffmpeg.org Date: Wed, 10 May 2023 15:55:09 +0200 Message-Id: <20230510135509.4074-6-ffmpeg@haasn.xyz> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230510135509.4074-1-ffmpeg@haasn.xyz> References: <20230510135509.4074-1-ffmpeg@haasn.xyz> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 6/6] lavfi/vf_libplacebo: add frame_mixer option X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Niklas Haas Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: sQ+agQhagof2 From: Niklas Haas Fairly straightforward. We just need to modify the scaler handling code slightly to support looking at a different list of filter presets. --- doc/filters.texi | 31 +++++++++++++++++++++++++++++++ libavfilter/vf_libplacebo.c | 17 +++++++++++------ 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 3e17ba1654..42ce313743 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -16167,6 +16167,31 @@ Cubic BC spline with parameters recommended by Mitchell and Netravali. Very little ringing. @end table +@item frame_mixer +Controls the kernel used for mixing frames temporally. The default value is +@code{none}, which disables frame mixing. For a full list of possible values, +pass @code{help} to this option. The most important values are: +@table @samp +@item none +Disables frame mixing, giving a result equivalent to "nearest neighbour" +semantics. + +@item oversample +Oversamples the input video to create a "Smooth Motion"-type effect: if an +output frame would exactly fall on the transition between two video frames, it +is blended according to the relative overlap. This is the recommended option +whenever preserving the original subjective appearance is desired. + +@item mitchell_clamp +Larger filter kernel that smoothly interpolates multiple frames in a manner +designed to eliminate ringing and other artefacts as much as possible. This is +the recommended option wherever maximum visual smoothness is desired. + +@item linear +Linear blend/fade between frames. Especially useful for constructing e.g. +slideshows. +@end table + @item lut_entries Configures the size of scaler LUTs, ranging from @code{1} to @code{256}. The default of @code{0} will pick libplacebo's internal default, typically @@ -16524,6 +16549,12 @@ Rescale input to fit into standard 1080p, with high quality scaling: libplacebo=w=1920:h=1080:force_original_aspect_ratio=decrease:normalize_sar=true:upscaler=ewa_lanczos:downscaler=ewa_lanczos @end example +@item +Interpolate low FPS / VFR input to smoothed constant 60 fps output: +@example +libplacebo=fps=60:frame_mixer=mitchell_clamp +@end example + @item Convert input to standard sRGB JPEG: @example diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index 2bec942358..9cbc1bac94 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -164,6 +164,7 @@ typedef struct LibplaceboContext { struct pl_render_params params; char *upscaler; char *downscaler; + char *frame_mixer; int lut_entries; float antiringing; int sigmoid; @@ -285,17 +286,19 @@ static int parse_shader(AVFilterContext *avctx, const void *shader, size_t len) static int find_scaler(AVFilterContext *avctx, const struct pl_filter_config **opt, - const char *name) + const char *name, int frame_mixing) { - const struct pl_filter_preset *preset; + const struct pl_filter_preset *preset, *presets_avail; + presets_avail = frame_mixing ? pl_frame_mixers : pl_scale_filters; + if (!strcmp(name, "help")) { av_log(avctx, AV_LOG_INFO, "Available scaler presets:\n"); - for (preset = pl_scale_filters; preset->name; preset++) + for (preset = presets_avail; preset->name; preset++) av_log(avctx, AV_LOG_INFO, " %s\n", preset->name); return AVERROR_EXIT; } - for (preset = pl_scale_filters; preset->name; preset++) { + for (preset = presets_avail; preset->name; preset++) { if (!strcmp(name, preset->name)) { *opt = preset->filter; return 0; @@ -411,8 +414,9 @@ static int update_settings(AVFilterContext *ctx) .disable_fbos = s->disable_fbos, ); - RET(find_scaler(ctx, &s->params.upscaler, s->upscaler)); - RET(find_scaler(ctx, &s->params.downscaler, s->downscaler)); + RET(find_scaler(ctx, &s->params.upscaler, s->upscaler, 0)); + RET(find_scaler(ctx, &s->params.downscaler, s->downscaler, 0)); + RET(find_scaler(ctx, &s->params.frame_mixer, s->frame_mixer, 1)); return 0; fail: @@ -1107,6 +1111,7 @@ static const AVOption libplacebo_options[] = { { "upscaler", "Upscaler function", OFFSET(upscaler), AV_OPT_TYPE_STRING, {.str = "spline36"}, .flags = DYNAMIC }, { "downscaler", "Downscaler function", OFFSET(downscaler), AV_OPT_TYPE_STRING, {.str = "mitchell"}, .flags = DYNAMIC }, + { "frame_mixer", "Frame mixing function", OFFSET(frame_mixer), AV_OPT_TYPE_STRING, {.str = "none"}, .flags = DYNAMIC }, { "lut_entries", "Number of scaler LUT entries", OFFSET(lut_entries), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 256, DYNAMIC }, { "antiringing", "Antiringing strength (for non-EWA filters)", OFFSET(antiringing), AV_OPT_TYPE_FLOAT, {.dbl = 0.0}, 0.0, 1.0, DYNAMIC }, { "sigmoid", "Enable sigmoid upscaling", OFFSET(sigmoid), AV_OPT_TYPE_BOOL, {.i64 = 1}, 0, 1, DYNAMIC },