| Message ID | DM8P223MB03650A5791C36AF9040F74C3BA749@DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM |
|---|---|
| Headers | show |
| Series | Subtitle Filtering | expand |
On Mon, Dec 13, 2021 at 11:40:16PM +0000, Soft Works wrote: > New in V24 > > - Fixes bugs as reported by Michal > - graphicsub2video: use 1x1 output frame size as long as subtitle size is unknown (0x0) and no size is not explicitly set > - decode: set subtitle frame size from decoding context > - ffmpeg: re-init graph when subtitle size changes > - ffmpeg_filter: always insert subscale filter for sub2video compatibility > - vf_overlaytextsubs: ensure equal in/out video formats > > GITHUB Branch: https://github.com/softworkz/FFmpeg/tree/subs_v24_plus I didnt immedeatly spot any "hard" failures from this. Quite a few cases change output, should i report these ? I didnt see anything obviously wrong vissually in any heres a random pick: ./ffmpeg -f lavfi -i "movie=tickets/1332/Starship_Troopers.vob[out0+subcc]" -vn -map s /tmp/file-eia608.srt @@ -6,12 +6,12 @@ 2 00:00:15,249 --> 00:00:18,252 <font face="Monospace">{\an7}- TAKE THE NUMBER TWO CHAIR, -\h\hIBANEZ. + IBANEZ. - YES, MA’AM.</font> 3 00:00:22,756 --> 00:00:27,761 -<font face="Monospace">{\an7}\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\hIDENTIFY. +<font face="Monospace">{\an7} IDENTIFY. IBANEZ, "T"-THREE-TWO-FIVE-"A," CLEAR.</font> [...]
> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Michael > Niedermayer > Sent: Tuesday, December 14, 2021 11:06 PM > To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> > Subject: Re: [FFmpeg-devel] [PATCH v24 00/21] Subtitle Filtering > > On Mon, Dec 13, 2021 at 11:40:16PM +0000, Soft Works wrote: > > New in V24 > > > > - Fixes bugs as reported by Michal > > - graphicsub2video: use 1x1 output frame size as long as subtitle size is > unknown (0x0) and no size is not explicitly set > > - decode: set subtitle frame size from decoding context > > - ffmpeg: re-init graph when subtitle size changes > > - ffmpeg_filter: always insert subscale filter for sub2video compatibility > > - vf_overlaytextsubs: ensure equal in/out video formats > > > > GITHUB Branch: https://github.com/softworkz/FFmpeg/tree/subs_v24_plus > > I didnt immedeatly spot any "hard" failures from this. > Quite a few cases change output, should i report these ? I didnt see anything > obviously wrong vissually in any Thanks again for testing! I think as long as the results are visually equal, we should be just fine. We already know that there are several categories which are causing binary differences: - The processing differences - sub2video always creates full RGBA frames which may require conversion while the new filters can overlay on yuva directly - when scaling is required, the new subscale filter scales the individual subtitle rects only to overlay directly or draw onto transparent full frames afterwards - Timing differences - Sometimes like one frame earlier or later (as can be seen in the test ref diffs) - Different eof behavior actually 'more correct' than before (see my previous response where the current implementation outputs an erroneous frame at the end with bogus timing) I've also seen a case once, where a 1-pixel offset exists between current and new implementation, but I don't think that this is actionable without having a case with a clear indication about which (current or new) is right and which is wrong, and whether a "right/wrong" even exist in such cases. Most likely this is caused by rounding differences that occur when scaling individual rects (and their integer positions) instead of full frames. So, from my side, we're all good, as long as there are no visible differences. > heres a random pick: > ./ffmpeg -f lavfi -i "movie=tickets/1332/Starship_Troopers.vob[out0+subcc]" > -vn -map s /tmp/file-eia608.srt > > @@ -6,12 +6,12 @@ > 2 > 00:00:15,249 --> 00:00:18,252 > <font face="Monospace">{\an7}- TAKE THE NUMBER TWO CHAIR, > -\h\hIBANEZ. > + IBANEZ. > - YES, MA’AM.</font> > > 3 > 00:00:22,756 --> 00:00:27,761 > -<font > face="Monospace">{\an7}\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\h\hIDENTIFY. > +<font face="Monospace">{\an7} IDENTIFY. > IBANEZ, "T"-THREE-TWO-FIVE-"A," > CLEAR.</font> Yes, this change is intentional and the result of the bugfix in commit > avutil/ass_split: Add parsing of hard-space tags (\h) (not on the ML, only on GitHub) Hard spaces in form of '\h' are specific to the ASS/SSA spec, but current ffmpeg doesn't parse and convert them. Those '\h' sequences are invalid in ttml, text and webvtt. For SRT, there's no clear spec and while I've seen it somewhere declared to be "valid", it is not supported by all clients, that's why I've changed this to emit spaces instead, just like for ttml and text. For webvtt I'm converting them to I think I've fond away how to submit those commits in a way that they can be applied by patchwork despite the mixed line endings. Will make another attempt shortly. Thanks again for testing, softworkz
New in V24 - Fixes bugs as reported by Michal - graphicsub2video: use 1x1 output frame size as long as subtitle size is unknown (0x0) and no size is not explicitly set - decode: set subtitle frame size from decoding context - ffmpeg: re-init graph when subtitle size changes - ffmpeg_filter: always insert subscale filter for sub2video compatibility - vf_overlaytextsubs: ensure equal in/out video formats GITHUB Branch: https://github.com/softworkz/FFmpeg/tree/subs_v24_plus without extras: https://github.com/softworkz/FFmpeg/commit/c50eaa9307d448670f7119db4df0a2534ec4a6c5 New in V23 - subfmt.h: Fix includes, use #if rather than #ifdef - subfmt.c: Remove unused struct - move addition of ass hard_space callback to commit 5 - also add ass extensibility callback - remove bugfix commits regarding ass hard-space due to being incompatible with GIT e-mail workflow New in V22 - Two new commits, implementing parsing of hard-space ASS tags (\h} The ccaption_dec decoder emits a lot of \h tags via splitcc which the encoders (all using ass_split) didn't handle properly - Combined APIchanges and version bumps into a single commit at the end - Moved new fields to end of AVFrame - Changed version bumps as suggested => no more major bumps now! New in V21 - Rebased. Patchset was broken again. - Don't declare AVFrame.subtitle_pts as deprecated and clarify doc text - Remove AVFilter.subs_list and AVFilter.sub_fmt from APIchanges - Change include in formats.c New in V20 - Rebased. V19 didn't apply cleanly anymore New in V19 - Document API changes in all relevant commits - Move enum AVSubtitleType: Improved commit message - Put deprecated enum values under removal guards - Document AV_SUBTITLE_FMT_NB - Fixed all trailing whitespace - splitcc: Add scatter_realtime_output option - splitcc: Emit empty subtitle frames instead of repeating - New commit: Replace deprecated enum values New in V18 - rebased to latest head (to fix merge conflict in allfilters.c) - textmod: Removed two trailing blanks from sf_textmod.c - xsubdec: fix xsubdec regression found by Michael; packet size estimation as flawed - splitcc: Fix uninitialzed locals - splitcc: Do not propagate hwcontext on secondary output - testsub2video: fix output timeing, re-add mutex locking and implement render_latest_only parameter New in V17 - Applied almost all suggestions that were made (thanks everybody!) - Split the initial commit, no longer moving the legacy structs (AVSubtitle..) - Use the regular encoding API for subtitles - Updated subtitle encoders to handle packet allocation - Removed compatibility layer for accessing legacy encoders - Updated the compatibility implementation for the legacy subtitle encode api - Reordered commits and merged ffmpeg.c changes into a single commit New in V16 - Added missing reference to subfmt.h iun libzvbi-teletextdec.c - Fixed Fate error in patch 15/16 - Removed all trsiling whitespace in tilers.texi New in V15 - Rebased to upstream changes - avcodec/subtitles: Migrate subtitle encoders to frame-based API and provide compatibility shim for legacy api - fftools/ffmpeg: Use new frame-based subtitle encoding API - AVSubtitleArea: copy flags field, make params const - graphicsubs2text: Don't emit duplicate frames - graphicsubs2text: Combined OCR output into a single AVSubtitleArea (I have a prototype for detecting text colors and positions, but it's not ready at this point) - splitcc: cleanup local subtitle_header ref - stripstyles: add parameter for ass layer selection - avcodec/subtitles: deferred loading of ass header for text subtitle encoders - verified all example command lines in the docs are working, added somre more Kind regards, softworkz softworkz (21): avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values avutil/frame: Prepare AVFrame for subtitle handling avcodec/subtitles: Introduce new frame-based subtitle decoding API avfilter/subtitles: Update vf_subtitles to use new decoding api avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing avcodec/subtitles: Migrate subtitle encoders to frame-based API and provide a compatibility shim for the legacy api avcodec/subtitles: Replace deprecated enum values fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters fftools/ffmpeg: Replace sub2video with subtitle frame filtering and use new frame-based subtitle encoding API avfilter/avfilter: Fix hardcoded input index avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/splitcc: Add splitcc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles doc/APIchanges: update for subtitle filtering changes configure | 7 +- doc/APIchanges | 23 + doc/filters.texi | 756 +++++++++++++++ fftools/ffmpeg.c | 588 ++++++------ fftools/ffmpeg.h | 15 +- fftools/ffmpeg_filter.c | 236 +++-- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 +- fftools/ffprobe.c | 48 +- libavcodec/Makefile | 56 +- libavcodec/ass.h | 147 +-- libavcodec/assdec.c | 4 +- libavcodec/assenc.c | 92 +- libavcodec/avcodec.h | 32 +- libavcodec/ccaption_dec.c | 19 +- libavcodec/codec_desc.c | 11 + libavcodec/codec_desc.h | 8 + libavcodec/decode.c | 56 +- libavcodec/dvbsubdec.c | 2 +- libavcodec/dvbsubenc.c | 96 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 100 +- libavcodec/encode.c | 63 +- libavcodec/internal.h | 16 + libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 124 ++- libavcodec/mpl2dec.c | 2 +- libavcodec/pgssubdec.c | 2 +- libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 112 ++- libavcodec/subviewerdec.c | 2 +- libavcodec/tests/avcodec.c | 2 - libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 111 ++- libavcodec/utils.c | 182 ++++ libavcodec/version.h | 2 +- libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 90 +- libavcodec/xsubdec.c | 2 +- libavcodec/xsubenc.c | 87 +- libavfilter/Makefile | 15 + libavfilter/allfilters.c | 13 + libavfilter/avfilter.c | 30 +- libavfilter/avfilter.h | 11 + libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 54 ++ libavfilter/buffersink.h | 7 + libavfilter/buffersrc.c | 72 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 22 + libavfilter/formats.h | 3 + libavfilter/internal.h | 19 +- libavfilter/sf_graphicsub2text.c | 354 +++++++ libavfilter/sf_splitcc.c | 378 ++++++++ libavfilter/sf_stripstyles.c | 196 ++++ libavfilter/sf_subscale.c | 883 ++++++++++++++++++ libavfilter/sf_textmod.c | 697 ++++++++++++++ libavfilter/subtitles.c | 63 ++ libavfilter/subtitles.h | 44 + libavfilter/vf_overlaygraphicsubs.c | 742 +++++++++++++++ libavfilter/vf_overlaytextsubs.c | 643 +++++++++++++ libavfilter/vf_subtitles.c | 54 +- libavutil/Makefile | 4 + {libavcodec => libavutil}/ass.c | 87 +- libavutil/ass_internal.h | 133 +++ {libavcodec => libavutil}/ass_split.c | 30 +- .../ass_split_internal.h | 32 +- libavutil/frame.c | 211 ++++- libavutil/frame.h | 78 +- libavutil/subfmt.c | 45 + libavutil/subfmt.h | 115 +++ libavutil/version.h | 3 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 181 ++-- tests/ref/fate/sub-dvb | 162 ++-- tests/ref/fate/sub2video | 220 ++--- tests/ref/fate/sub2video_basic | 135 +-- tests/ref/fate/sub2video_time_limited | 4 +- 84 files changed, 7572 insertions(+), 1416 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_splitcc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlaygraphicsubs.c create mode 100644 libavfilter/vf_overlaytextsubs.c rename {libavcodec => libavutil}/ass.c (65%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (94%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h