mbox

[FFmpeg-devel,v2,0/8] A New Take on the Immortal Sub2Video Code

Message ID MN2PR04MB598180B4B211C545C33D184EBACB9@MN2PR04MB5981.namprd04.prod.outlook.com
Headers show

Message

Soft Works Aug. 30, 2021, 8:15 a.m. UTC 
v2 Update:

- Implemented Andreas' suggestions
- overlay_subs filter:
  - removed duplicated code
  - implemented direct (no pre-conversion) blending of graphical
    subtitle rects
  - Supported input formats:
    - all packed RGB formats (with and without alpha)
	- yuv420p, yuv422p, yuv444p


This patchset is about introducing filtering support for subtitles.

The current sub2video "hack" implementation is slow and ineffective: 
it creates a full-size video frame from the source subtitles 
and that full-size frame needs to be overlaid/blended with/onto every 
single video frame. That's a pretty expensive operation and it needs to be 
performed for every single video frame, even when there's nothing to 
overlay at all (= no subs to show in current scene).
And when there are subtitles to show, those are usually limited to smaller 
areas on the screen; again, there's no need to overlay a full-size image 
onto each frame. From a performance perspective, it would be much more 
efficient to overlay the individual rects onto each frame only instead of 
a full frame blend.

From a general perspective - even though it has always been considered as 
a 'hack' - the sub2video implementation has become a kind of standard 
behavior, despite of its shortcomings. Users have accommodated with the 
respective command lines and are expecting these to work. I'd expect any 
deviation from the current behavior to be filed as regression bugs.

From there, I had decided to go for a small-step improvement of that 
situation while staying inside these corner points, and the plan was simple:

- keep the current sub2video behavior (specifically hearbeat) working
  to avoid regressions
- but stop creating full video frames (with alpha) for feeding into a 
  filtergraph (and later overlay)
- instead, use refcounting for AVSubtitle and attach it to frames of (new) 
  type AVMediaType_Subtitle
- those subtitle frames can travel through a filtergraph without needing 
  to materialize them to images first

For more efficient overlay (processing the relevant rects only), I have 
created a new filter: overlay_subs:

- Input0: video
- Input1:  subtitles (format: SUBTITLE_BITMAP)
- Output0: video

In order to keep compatibility with existing command lines, I have added 
another filter: sub2video (reminescent naming)
sub2video creates video frames (with alpha) from an existing subtitle 
stream:

- Input0:  subrtitles (format: SUBTITLE_BITMAP)
- Output0: video

This filter gets auto-inserted to retain compatibility with current 
sub2video command lines.

As AVSubtitle can carry both, textual and graphical subtitles, the ability 
for sending textual subtitles through a filtergraph came more or less for 
free.
For testing purposes, I have added another filter: sleet (translates 
subtitles to 'L337 speak')

- Input0: subtitles (format: SUBTITLE_ASS or SUBTITLE_TEXT)
- Output0: subtitles (format: same as input)

Why leet? Real world use is surely questionable, but it has two advantages 
that make it a great choice for testing: 
You can see from almost every single line whether it has been filtered, 
and the text is still readable which allows to verify that timings are 
still valid.


Working Command Lines

Using "overlay_subs" (better performance):

-y -loglevel verbose -ss 00:02:30 -i INPUT -filter_complex 
"[0:0][0:3]overlay_subs=x=0:y=-220" output.mp4

Using regular Overlay and the new sub2video filter:

-y -loglevel verbose -ss 00:02:30 -filter_threads 1 -i INPUT 
-filter_complex 
"[0:0]format=bgra[main];[0:3]sub2video[subs];[main][subs]overlay=x=0:y=-220:
format=auto,format=yuv420p" output.mp4

Legacy command line compatibility:
(using regular overlay; sub2video filter is inserted automatically)

-y -loglevel verbose -ss 00:04:30 -filter_threads 1 -i 
INPUT -filter_complex 
"[0:0][0:3]overlay=x=0:y=-220:format=auto,format=yuv420p" output.mp4
 
 
To make this work, I roughly did the following:

- Added a new property 'type' to AVFrame of type enum AVMediaType
- Changed code that uses width/height for checking whether a frame is 
  video or audio and check the 'type' property instead
- Added ff_ssrc_sbuffer and ff_ssink_sbuffer filters
- Changed filtergraph code to support frames with AVMediaType == 
  AVMediaType_Subtitle
- Transformed the existing sub2video code accordingly, while still keeping 
the heartbeat mechanism in place


Next Steps

- Create modified version of the ASS filter that works based on (text) 
 subtitle filter input
- Create modified EIA-608/708 filter (e.g. split_eia608) which has one 
  video input and one video output, plus a second output of type 'subtitle' 
 (format: text or ass)
  This will allow to burn-in eia subs or save them in any other subtitle 
  format without needing the weird "movie:xxx" input filter construct.

Regards,
softworkz
---

softworkz (8):
  lavu/frame: avframe add type property
  avfilter/subtitles: Add subtitles.c
  avfilter/avfilter: Handle subtitle frames
  avfilter/overlay_subs: Add overlay_subs filter
  avfilter/sub2video: Add sub2video filter
  avfilter/sbuffer: Add sbuffersrv and sbuffersink filters
  avfilter/sleet: Add sleet filter
  fftools/ffmpeg: Replace sub2video with subtitle frame filtering

 configure                     |   2 +-
 fftools/ffmpeg.c              | 324 +++++++++++---------
 fftools/ffmpeg.h              |   7 +-
 fftools/ffmpeg_filter.c       | 198 ++++++++----
 fftools/ffmpeg_hw.c           |   2 +-
 fftools/ffmpeg_opt.c          |   3 +-
 libavfilter/Makefile          |   6 +
 libavfilter/allfilters.c      |  13 +-
 libavfilter/avfilter.c        |   8 +-
 libavfilter/avfiltergraph.c   |   5 +
 libavfilter/buffersink.c      |  65 ++++
 libavfilter/buffersink.h      |  15 +
 libavfilter/buffersrc.c       |  65 ++++
 libavfilter/buffersrc.h       |   1 +
 libavfilter/f_interleave.c    |   3 +
 libavfilter/formats.c         |  11 +
 libavfilter/internal.h        |   1 +
 libavfilter/sf_sleet.c        | 209 +++++++++++++
 libavfilter/subtitles.c       |  61 ++++
 libavfilter/subtitles.h       |  44 +++
 libavfilter/svf_sub2video.c   | 260 ++++++++++++++++
 libavfilter/vf_overlay_subs.c | 551 ++++++++++++++++++++++++++++++++++
 libavfilter/vf_overlay_subs.h |  65 ++++
 libavutil/frame.c             |  74 ++++-
 libavutil/frame.h             |  39 ++-
 libavutil/version.h           |   2 +-
 26 files changed, 1804 insertions(+), 230 deletions(-)
 create mode 100644 libavfilter/sf_sleet.c
 create mode 100644 libavfilter/subtitles.c
 create mode 100644 libavfilter/subtitles.h
 create mode 100644 libavfilter/svf_sub2video.c
 create mode 100644 libavfilter/vf_overlay_subs.c
 create mode 100644 libavfilter/vf_overlay_subs.h