From patchwork Thu Aug 19 14:53:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas George X-Patchwork-Id: 29615 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6602:2a4a:0:0:0:0 with SMTP id k10csp372535iov; Thu, 19 Aug 2021 07:53:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxO+vuVGwRXo6aEjElqvQ1rTA2JaRmhLjcwRIzHtalKTTkRD7ZLfV0/CoAnDBNr4hJzmgiK X-Received: by 2002:a17:906:3fd7:: with SMTP id k23mr16153410ejj.176.1629384838528; Thu, 19 Aug 2021 07:53:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629384838; cv=none; d=google.com; s=arc-20160816; b=vN/aNnEubv2ft7AKk2JJ/u+H4lqnUliZhKCjiQj9MngWLxWQ3Ab8jTwzH6SN3QKyao 2M2l/1Px/B0jwLrBtJ4trFqBo7PmF4Ihr9HSQiTx02fa1A3tQnTNtxB5idMjy1J/5Fbk RfntAg81zepk2wb3VDyJFJteBjOU/BOIWJep5s0a7DKkMeTvMYxtMryLm9674pUiv92n uY2npp8yjQhLDiUvau+Q/rEqTdDI0YJs3ece0gzjKblUp5Ki5Xja9Gcak9RfsWev8ady b1A9sFUv6DiV58cO9Oi63myxqd3EH70Zor4hTAfCh4JXCr5CLBTv0VVJb17+QQEv036k U0BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :delivered-to; bh=mpbN745s3TU1LJNleXvu551+i04r8aM5bxCpvsdHwQk=; b=BLJIT6bMNd5E+7OPkKdFECqaA87lIN8+2SP5UnhXlt6NgY0oVeLYvpyTB2GQ/O2p4Q 83UpSlV54nH5wbs/OKB0ti2OMUx0baAmiMuOUfKPTSRbk2LMDPV3HZaWmNd28uNA9mrb kMndf0F9VXOO/05GXc0lich97ApiQ7uV+hWZwlPqgMK+JbxGkL3XiXmCUBwCkx8DL1mI GX7uSVB17qa7QPcpguibLXBLDxkxGruZlIfZsHEu3vruqQvwdFwaf6JPx5/0I2YNk0kc i/bEoa3sv1F5ZAWKYn/G7lRG+ciU2yqOI1khKbc6GRo5zNxRMGBcgt0fLwsUy9Cay3/s 4Y5A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id me14si3226534ejb.611.2021.08.19.07.53.56; Thu, 19 Aug 2021 07:53:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3330968A1B9; Thu, 19 Aug 2021 17:53:53 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from nef.ens.fr (nef2.ens.fr [129.199.96.40]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 505D2689BC9 for ; Thu, 19 Aug 2021 17:53:46 +0300 (EEST) X-ENS-nef-client: 129.199.129.80 ( name = phare.normalesup.org ) Received: from phare.normalesup.org (phare.normalesup.org [129.199.129.80]) by nef.ens.fr (8.14.4/1.01.28121999) with ESMTP id 17JErjZI017421 for ; Thu, 19 Aug 2021 16:53:45 +0200 Received: by phare.normalesup.org (Postfix, from userid 1001) id 34CD5EB5BC; Thu, 19 Aug 2021 16:53:45 +0200 (CEST) From: Nicolas George To: ffmpeg-devel@ffmpeg.org Date: Thu, 19 Aug 2021 16:53:42 +0200 Message-Id: <20210819145342.104364-1-george@nsup.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nef.ens.fr [129.199.96.32]); Thu, 19 Aug 2021 16:53:45 +0200 (CEST) Subject: [FFmpeg-devel] [PATCH] lavfi/formats: document the negotiation process. X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6WB49EqSZ3qf Signed-off-by: Nicolas George --- libavfilter/formats.h | 85 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/libavfilter/formats.h b/libavfilter/formats.h index ed513c265a..b3e780a41d 100644 --- a/libavfilter/formats.h +++ b/libavfilter/formats.h @@ -75,6 +75,91 @@ typedef struct AVFilterFormatMerger { int (*can_merge)(const void *a, const void *b); } AVFilterFormatsMerger; +/** + * Callbacks and properties to describe the steps of a format negotiation. + * + * The steps are: + * + * 1. query_formats(): call the callbacks on all filter to set lists of + * supported formats. + * When links on a filter must eventually have the same + * format, the lists of supported formats are the same + * object in memory. + * See: + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#12 + * + * + * 2. query_formats(): merge lists of supported formats or insert automatic + * conversion filters. + * Compute the intersection of the lists of supported + * formats on the ends of links. If it succeeds, replace + * both objects with the intersection everywhere they + * are referenced. + * If the intersection is empty, insert an automatic + * conversion filter. + * If several formats are negotiated at once (format, + * rate, layout), only merge if all three can be, since + * the conversion filter can convert all three at once. + * This process goes on as long as progress is made. + * See: + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#14 + * http://www.normalesup.org/~george/articles/format_negotiation_in_libavfilter/#29 + * + * 3. reduce_formats(): try to reduce format conversion within filters. + * For each link where there is only one supported + * formats on output, for each output of the connected + * filter, if the media type is the same and said + * format is supported, keep only this one. + * This process goes on as long as progress is made. + * Rationale: conversion filters will set a large list + * of supported formats on outputs but users will + * expect the output to be as close as possible as the + * input (examples: scale without changing the pixel + * format, resample without changint the layout). + * FIXME: this can probably be done by merging the + * input and output lists instead of re-implementing + * the logic. + * + * 4. swap_sample_fmts(): + * swap_samplerates(): + * swap_channel_layouts(): For each filter with an input with only one + * supported format, when outputs have several + * supported formats, put the best one with + * reference to the input at the beginning of the + * list, to prepare it for being picked up by + * pick_formats(). + * The best format is the one that is most + * similar to the input while not losing too much + * information. + * This process need to run only once. + * FIXME: reduce_formats() operates on all inputs + * with a single format, swap_*() operates on the + * first one only: check if the difference makes + * sense. + * TODO: the swapping done for one filter can + * override the swapping done for another filter + * connected to the same list of formats, maybe + * it would be better to compute a total score + * for all connected filters and use the score to + * pick the format instead of just swapping. + * TODO: make the similarity logic available as + * public functions in libavutil. + * + * 5. pick_formats(): Choose one format from the lists of supported formats, + * use it for the link and reduce the list to a single + * element to force other filters connected to the same + * list to use it. + * First process all links where there is a single format + * and the output links of all filters with an input, + * trying to preserve similarity between input and + * outputs. + * Repeat as long as process is made. + * Then do a final run for the remaining filters. + * FIXME: the similarity logic (the ref argument to + * pick_format()) added in FFmpeg duplicates and + * overrides the swapping logic added in libav. Better + * merge them into a score system. + */ typedef struct AVFilterNegotiation { unsigned nb; const AVFilterFormatsMerger *mergers;