From patchwork Fri Jan 5 16:42:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 45495 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1093:b0:199:52ba:a72d with SMTP id w19csp213293pze; Fri, 5 Jan 2024 08:44:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IE3ua7MrgUva/WJgoDpRnsHyodwyRadwVBb+uxqeZwPYVaZBxK76LueqOAaKvsEJgjtEnyp X-Received: by 2002:a17:906:74c1:b0:a28:fab0:9004 with SMTP id z1-20020a17090674c100b00a28fab09004mr799944ejl.86.1704473054122; Fri, 05 Jan 2024 08:44:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704473054; cv=none; d=google.com; s=arc-20160816; b=LT3wKawouUkP1sxHWvEaqLAPrTPfMd24nvWc5QF5396BtQsr3En2Fl/fLNX7Hyh0BL 7k+AFj53D+IA24kOfo0jbhEoeL7jrIh583Y7XHIX1ZX3bwTg/cMKIfYnN6z7Z66mtn0l bLYUJSm73dOKAx+CHY/y4cG0GDRzzKNmtS8wPqbVfwrRgatLvJwjeZL1/0P+i9cUXx0b iEvmajy8AiPt6KHycZc4pX9NBALmtE5dc4p9j001d4RUvgnkBD0fOdhm9hxGF1kBpBIM W5hd7dUtvKRk/6/2WDx8M5rodtttfcg6vsWeXaFF00/Mpc5spx8YnkjScgsWJ1BMwVan n5lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=vHC6OWKTRRnLZNXcDvuyy0TLgbhjEEZsk4aJVBBMkCg=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=bBaYER3Jg/dmIh7zoQyEXWYcilJ1/utWBtBVVjGhuNlnMweu3T0PB6y4RXy7OGrteh 5u3Gsm5O/tMPwzy84Zd0JFf/UmmVJWcp/OimM57j9sNYs89kStq3XisOJcEBWGPfYRGF G3BGLdlCkKRq6wrrgWLSmuQ90zU35gCIPwE43+YtAwfE884HSib4MSVWFe+nb7fkQ9Qj PUuRUJU9DOIp/u9UJ+MpydpjtMTE94XtZT4fxLtlaonHZ+Jp6YIm93omh9IaIcL5bh7z q6g2R6NyBRoDmxY/2qsK5Y2knK/2XAD1LkhCYhpv2xG4MDJipsJtsPJpLBLEZXi1vRY5 iidQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@khirnov.net header.s=mail header.b=kBsZ4egJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id gh38-20020a1709073c2600b00a2934eecdcesi622585ejc.425.2024.01.05.08.44.13; Fri, 05 Jan 2024 08:44:14 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@khirnov.net header.s=mail header.b=kBsZ4egJ; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DCAE268CEBC; Fri, 5 Jan 2024 18:43:11 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail1.khirnov.net (quelana.khirnov.net [94.230.150.81]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EA8AA68CDFB for ; Fri, 5 Jan 2024 18:43:00 +0200 (EET) Authentication-Results: mail1.khirnov.net; dkim=pass (2048-bit key; unprotected) header.d=khirnov.net header.i=@khirnov.net header.a=rsa-sha256 header.s=mail header.b=kBsZ4egJ; dkim-atps=neutral Received: from localhost (mail1.khirnov.net [IPv6:::1]) by mail1.khirnov.net (Postfix) with ESMTP id 5465A1579 for ; Fri, 5 Jan 2024 17:42:58 +0100 (CET) Received: from mail1.khirnov.net ([IPv6:::1]) by localhost (mail1.khirnov.net [IPv6:::1]) (amavis, port 10024) with ESMTP id PhIrRMzXYEP5 for ; Fri, 5 Jan 2024 17:42:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=khirnov.net; s=mail; t=1704472974; bh=0Om7Dk+u8MYnEHOXdswZRM7PeJ2Ongph7PLDZC1SyI4=; h=From:To:Subject:Date:In-Reply-To:References:From; b=kBsZ4egJb4dYov1wJUXUxCCj+rvq+Uoj9CByQZTtkCFJRxev/m/c/mdNkiLUdknX/ u3osw5r43ySBAai308qCESi/l6Kr94iz1MVPNbNJE4zK35cu2va6P9wxNzHgmfpLdg OnP8GHAoIgqVVB3C1ehSKJLEirLzw7a+7OnG4y4JEM2GeXNhDLOLjaao/dlB26YJ19 jYxXVVbvbafpflrTSsR24JRgifsHDTgVYEw8mA1KnmfBRtL9l6ajGQIlBnATWhTF/Y Ma8/DVfaKHov7HBJvPwmIw46bVjRthXBwkX2VnoPyPR9rd3PnkhJZaTRu+rMRPKXwi fLYo4WOO1v74Q== Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail1.khirnov.net (Postfix) with ESMTPS id F3F1A19B7 for ; Fri, 5 Jan 2024 17:42:53 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id 8F3B33A07D6 for ; Fri, 5 Jan 2024 17:42:53 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Fri, 5 Jan 2024 17:42:50 +0100 Message-ID: <20240105164251.28935-7-anton@khirnov.net> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240105164251.28935-1-anton@khirnov.net> References: <20240105164251.28935-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 7/8] fftools/ffmpeg_demux: implement -bsf for input X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: bp2sUmF2pr7F Previously bitstream filters could only be applied right before muxing, this allows to apply them right after demuxing. --- Changelog | 1 + doc/ffmpeg.texi | 22 +++-- fftools/ffmpeg_demux.c | 139 ++++++++++++++++++++++++++++---- fftools/ffmpeg_opt.c | 2 +- tests/fate/ffmpeg.mak | 5 ++ tests/ref/fate/ffmpeg-bsf-input | 18 +++++ 6 files changed, 164 insertions(+), 23 deletions(-) create mode 100644 tests/ref/fate/ffmpeg-bsf-input diff --git a/Changelog b/Changelog index 5b2899d05b..f8191d88c7 100644 --- a/Changelog +++ b/Changelog @@ -18,6 +18,7 @@ version : - lavu/eval: introduce randomi() function in expressions - VVC decoder - fsync filter +- ffmpeg CLI -bsf option may now be used for input as well as output version 6.1: - libaribcaption decoder diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi index d75517b443..59f7badcb6 100644 --- a/doc/ffmpeg.texi +++ b/doc/ffmpeg.texi @@ -2093,26 +2093,34 @@ an output mpegts file: ffmpeg -i inurl -streamid 0:33 -streamid 1:36 out.ts @end example -@item -bsf[:@var{stream_specifier}] @var{bitstream_filters} (@emph{output,per-stream}) -Apply bitstream filters to matching streams. +@item -bsf[:@var{stream_specifier}] @var{bitstream_filters} (@emph{input/output,per-stream}) +Apply bitstream filters to matching streams. The filters are applied to each +packet as it is received from the demuxer (when used as an input option) or +before it is sent to the muxer (when used as an output option). @var{bitstream_filters} is a comma-separated list of bitstream filter -specifications. The specified bitstream filters are applied to coded packets in -the order they are written in. Each bitstream filter specification is of the -form +specifications, each of the form @example @var{filter}[=@var{optname0}=@var{optval0}:@var{optname1}=@var{optval1}:...] @end example Any of the ',=:' characters that are to be a part of an option value need to be escaped with a backslash. -Use the @code{-bsfs} option to get the list of bitstream filters. +Use the @code{-bsfs} option to get the list of bitstream filters. E.g. @example -ffmpeg -i h264.mp4 -c:v copy -bsf:v h264_mp4toannexb -an out.h264 +ffmpeg -bsf:v h264_mp4toannexb -i h264.mp4 -c:v copy -an out.h264 @end example +applies the @code{h264_mp4toannexb} bitstream filter (which converts +MP4-encapsulated H.264 stream to Annex B) to the @emph{input} video stream. + +On the other hand, @example ffmpeg -i file.mov -an -vn -bsf:s mov2textsub -c:s copy -f rawvideo sub.txt @end example +applies the @code{mov2textsub} bitstream filter (which extracts text from MOV +subtitles) to the @emph{output} subtitle stream. Note, however, that since both +examples use @code{-c copy}, it matters little whether the filters are applied +on input or output - that would change if transcoding was hapenning. @item -tag[:@var{stream_specifier}] @var{codec_tag} (@emph{input/output,per-stream}) Force a tag/fourcc for matching streams. diff --git a/fftools/ffmpeg_demux.c b/fftools/ffmpeg_demux.c index eae1f0bde5..16d4f67e59 100644 --- a/fftools/ffmpeg_demux.c +++ b/fftools/ffmpeg_demux.c @@ -34,6 +34,7 @@ #include "libavutil/time.h" #include "libavutil/timestamp.h" +#include "libavcodec/bsf.h" #include "libavcodec/packet.h" #include "libavformat/avformat.h" @@ -71,6 +72,8 @@ typedef struct DemuxStream { const AVCodecDescriptor *codec_desc; + AVBSFContext *bsf; + /* number of packets successfully read for this stream */ uint64_t nb_packets; // combined size of all the packets read @@ -118,6 +121,8 @@ typedef struct Demuxer { typedef struct DemuxThreadContext { // packet used for reading from the demuxer AVPacket *pkt_demux; + // packet for reading from BSFs + AVPacket *pkt_bsf; } DemuxThreadContext; static DemuxStream *ds_from_ist(InputStream *ist) @@ -513,13 +518,17 @@ static int do_send(Demuxer *d, DemuxStream *ds, AVPacket *pkt, unsigned flags, return 0; } -static int demux_send(Demuxer *d, DemuxStream *ds, AVPacket *pkt, unsigned flags) +static int demux_send(Demuxer *d, DemuxThreadContext *dt, DemuxStream *ds, + AVPacket *pkt, unsigned flags) { InputFile *f = &d->f; int ret; + // pkt can be NULL only when flushing BSFs + av_assert0(ds->bsf || pkt); + // send heartbeat for sub2video streams - if (d->pkt_heartbeat && pkt->pts != AV_NOPTS_VALUE) { + if (d->pkt_heartbeat && pkt && pkt->pts != AV_NOPTS_VALUE) { for (int i = 0; i < f->nb_streams; i++) { DemuxStream *ds1 = ds_from_ist(f->streams[i]); @@ -537,10 +546,69 @@ static int demux_send(Demuxer *d, DemuxStream *ds, AVPacket *pkt, unsigned flags } } - ret = do_send(d, ds, pkt, flags, "demuxed"); - if (ret < 0) - return ret; + if (ds->bsf) { + if (pkt) + av_packet_rescale_ts(pkt, pkt->time_base, ds->bsf->time_base_in); + ret = av_bsf_send_packet(ds->bsf, pkt); + if (ret < 0) { + if (pkt) + av_packet_unref(pkt); + av_log(ds, AV_LOG_ERROR, "Error submitting a packet for filtering: %s\n", + av_err2str(ret)); + return ret; + } + + while (1) { + ret = av_bsf_receive_packet(ds->bsf, dt->pkt_bsf); + if (ret == AVERROR(EAGAIN)) + return 0; + else if (ret < 0) { + if (ret != AVERROR_EOF) + av_log(ds, AV_LOG_ERROR, + "Error applying bitstream filters to a packet: %s\n", + av_err2str(ret)); + return ret; + } + + dt->pkt_bsf->time_base = ds->bsf->time_base_out; + + ret = do_send(d, ds, dt->pkt_bsf, 0, "filtered"); + if (ret < 0) { + av_packet_unref(dt->pkt_bsf); + return ret; + } + } + } else { + ret = do_send(d, ds, pkt, flags, "demuxed"); + if (ret < 0) + return ret; + } + + return 0; +} + +static int demux_bsf_flush(Demuxer *d, DemuxThreadContext *dt) +{ + InputFile *f = &d->f; + int ret; + + for (unsigned i = 0; i < f->nb_streams; i++) { + DemuxStream *ds = ds_from_ist(f->streams[i]); + + if (!ds->bsf) + continue; + + ret = demux_send(d, dt, ds, NULL, 0); + ret = (ret == AVERROR_EOF) ? 0 : (ret < 0) ? ret : AVERROR_BUG; + if (ret < 0) { + av_log(ds, AV_LOG_ERROR, "Error flushing BSFs: %s\n", + av_err2str(ret)); + return ret; + } + + av_bsf_flush(ds->bsf); + } return 0; } @@ -573,6 +641,7 @@ static void thread_set_name(InputFile *f) static void demux_thread_uninit(DemuxThreadContext *dt) { av_packet_free(&dt->pkt_demux); + av_packet_free(&dt->pkt_bsf); memset(dt, 0, sizeof(*dt)); } @@ -585,6 +654,10 @@ static int demux_thread_init(DemuxThreadContext *dt) if (!dt->pkt_demux) return AVERROR(ENOMEM); + dt->pkt_bsf = av_packet_alloc(); + if (!dt->pkt_bsf) + return AVERROR(ENOMEM); + return 0; } @@ -619,10 +692,22 @@ static void *input_thread(void *arg) continue; } if (ret < 0) { + int ret_bsf; + + if (ret == AVERROR_EOF) + av_log(d, AV_LOG_VERBOSE, "EOF while reading input\n"); + else { + av_log(d, AV_LOG_ERROR, "Error during demuxing: %s\n", + av_err2str(ret)); + ret = exit_on_error ? ret : 0; + } + + ret_bsf = demux_bsf_flush(d, &dt); + ret = err_merge(ret == AVERROR_EOF ? 0 : ret, ret_bsf); + if (d->loop) { /* signal looping to our consumers */ dt.pkt_demux->stream_index = -1; - ret = sch_demux_send(d->sch, f->index, dt.pkt_demux, 0); if (ret >= 0) ret = seek_to_start(d, (Timestamp){ .ts = dt.pkt_demux->pts, @@ -633,14 +718,6 @@ static void *input_thread(void *arg) /* fallthrough to the error path */ } - if (ret == AVERROR_EOF) - av_log(d, AV_LOG_VERBOSE, "EOF while reading input\n"); - else { - av_log(d, AV_LOG_ERROR, "Error during demuxing: %s\n", - av_err2str(ret)); - ret = exit_on_error ? ret : 0; - } - break; } @@ -677,7 +754,7 @@ static void *input_thread(void *arg) if (d->readrate) readrate_sleep(d); - ret = demux_send(d, ds, dt.pkt_demux, send_flags); + ret = demux_send(d, &dt, ds, dt.pkt_demux, send_flags); if (ret < 0) break; } @@ -735,9 +812,11 @@ static void demux_final_stats(Demuxer *d) static void ist_free(InputStream **pist) { InputStream *ist = *pist; + DemuxStream *ds; if (!ist) return; + ds = ds_from_ist(ist); dec_free(&ist->decoder); @@ -749,6 +828,8 @@ static void ist_free(InputStream **pist) avcodec_free_context(&ist->dec_ctx); avcodec_parameters_free(&ist->par); + av_bsf_free(&ds->bsf); + av_freep(pist); } @@ -1039,6 +1120,7 @@ static int ist_add(const OptionsContext *o, Demuxer *d, AVStream *st) const char *hwaccel = NULL; char *hwaccel_output_format = NULL; char *codec_tag = NULL; + char *bsfs = NULL; char *next; char *discard_str = NULL; int ret; @@ -1258,6 +1340,33 @@ static int ist_add(const OptionsContext *o, Demuxer *d, AVStream *st) return ret; } + MATCH_PER_STREAM_OPT(bitstream_filters, str, bsfs, ic, st); + if (bsfs) { + ret = av_bsf_list_parse_str(bsfs, &ds->bsf); + if (ret < 0) { + av_log(ist, AV_LOG_ERROR, + "Error parsing bitstream filter sequence '%s': %s\n", + bsfs, av_err2str(ret)); + return ret; + } + + ret = avcodec_parameters_copy(ds->bsf->par_in, ist->par); + if (ret < 0) + return ret; + ds->bsf->time_base_in = ist->st->time_base; + + ret = av_bsf_init(ds->bsf); + if (ret < 0) { + av_log(ist, AV_LOG_ERROR, "Error initializing bitstream filters: %s\n", + av_err2str(ret)); + return ret; + } + + ret = avcodec_parameters_copy(ist->par, ds->bsf->par_out); + if (ret < 0) + return ret; + } + ds->codec_desc = avcodec_descriptor_get(ist->par->codec_id); return 0; diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c index c189cf373b..76b50c0bad 100644 --- a/fftools/ffmpeg_opt.c +++ b/fftools/ffmpeg_opt.c @@ -1919,7 +1919,7 @@ const OptionDef options[] = { "0 = use frame rate (video) or sample rate (audio)," "-1 = match source time base", "ratio" }, - { "bsf", OPT_TYPE_STRING, OPT_SPEC | OPT_EXPERT | OPT_OUTPUT, + { "bsf", OPT_TYPE_STRING, OPT_SPEC | OPT_EXPERT | OPT_OUTPUT | OPT_INPUT, { .off = OFFSET(bitstream_filters) }, "A comma-separated list of bitstream filters", "bitstream_filters", }, diff --git a/tests/fate/ffmpeg.mak b/tests/fate/ffmpeg.mak index 1bfd5c1b31..df955df4d0 100644 --- a/tests/fate/ffmpeg.mak +++ b/tests/fate/ffmpeg.mak @@ -256,3 +256,8 @@ FATE_SAMPLES_FFMPEG-$(call FRAMECRC, MPEGVIDEO, MPEG2VIDEO) += fate-ffmpeg-input fate-ffmpeg-error-rate-fail: CMD = ffmpeg -i $(TARGET_SAMPLES)/mkv/h264_tta_undecodable.mkv -c:v copy -f null -; test $$? -eq 69 fate-ffmpeg-error-rate-pass: CMD = ffmpeg -i $(TARGET_SAMPLES)/mkv/h264_tta_undecodable.mkv -c:v copy -f null - -max_error_rate 1 FATE_SAMPLES_FFMPEG-$(call ENCDEC, PCM_S16LE TTA, NULL MATROSKA) += fate-ffmpeg-error-rate-fail fate-ffmpeg-error-rate-pass + +# test input -bsf +# use -stream_loop, because it tests flushing bsfs +fate-ffmpeg-bsf-input: CMD = framecrc -stream_loop 2 -bsf setts=PTS*2 -i $(TARGET_SAMPLES)/hevc/extradata-reload-multi-stsd.mov -c copy +FATE_SAMPLES_FFMPEG-$(call FRAMECRC, MOV, , SETTS_BSF) += fate-ffmpeg-bsf-input diff --git a/tests/ref/fate/ffmpeg-bsf-input b/tests/ref/fate/ffmpeg-bsf-input new file mode 100644 index 0000000000..67dd57cf6d --- /dev/null +++ b/tests/ref/fate/ffmpeg-bsf-input @@ -0,0 +1,18 @@ +#extradata 0: 110, 0xb4031479 +#tb 0: 1/25 +#media_type 0: video +#codec_id 0: hevc +#dimensions 0: 128x128 +#sar 0: 1/1 +0, 0, 0, 1, 2108, 0x57c38f64 +0, 2, 2, 1, 31, 0xabe10d25, F=0x0 +0, 4, 4, 1, 1915, 0xd430347f, S=1, 109 +0, 6, 6, 1, 35, 0xc4ad0d4c, F=0x0 +0, 8, 8, 1, 2108, 0x57c38f64, S=1, 110 +0, 10, 10, 1, 31, 0xabe10d25, F=0x0 +0, 12, 12, 1, 1915, 0xd430347f, S=1, 109 +0, 14, 14, 1, 35, 0xc4ad0d4c, F=0x0 +0, 16, 16, 1, 2108, 0x57c38f64, S=1, 110 +0, 18, 18, 1, 31, 0xabe10d25, F=0x0 +0, 20, 20, 1, 1915, 0xd430347f, S=1, 109 +0, 22, 22, 1, 35, 0xc4ad0d4c, F=0x0