mbox series

[FFmpeg-devel,v4,00/23] Subtitle Filtering 2022

Message ID pull.18.v4.ffstaging.FFmpeg.1653744323.ffmpegagent@gmail.com
Headers show
Series Subtitle Filtering 2022 | expand

Message

softworkz May 28, 2022, 1:25 p.m. UTC
Subtitle Filtering 2022
=======================

This is a substantial update to the earlier subtitle filtering patch series.
A primary goal has been to address others' concerns as much as possible on
one side and to provide more clarity and control over the way things are
working. Clarity is is specifically important to allow for a better
understanding of the need for a subtitle start pts value that can be
different from the frame's pts value. This is done by refactoring the
subtitle timing fields in AVFrame, adding a frame field to indicate repeated
subtitle frames, and finally the full removal of the heartbeat
functionality, replaced by a new 'subfeed' filter that provides different
modes for arbitrating subtitle frames in a filter graph. Finally, each
subtitle filter's documentation has been amended by a section describing the
filter's timeline behavior (in v3 update).

The update also includes major improvements to graphicsub2text and lots of
other details.

Versioning is restarting at v1 due to the new submission procedure.


v4 - Quality Improvements
=========================

 * finally an updated version
 * includes many improvements from internal testing
 * all FATE tests passed
 * all example commands from the docs verified to work
 * can't list all the detail changes..
 * I have left out the extra commits which can be handled separately, just
   in case somebody wonders why these are missing:
   * avcodec/webvttenc: Don't encode drawing codes and empty lines
   * avcodec/webvttenc: convert hard-space tags to  
   * avutil/ass_split: Add parsing of hard-space tags (\h)
   * avutil/ass_split: Treat all content in curly braces as hidden
   * avutil/ass_split: Fix ass parsing of style codes with comments


v3 - Rebase
===========

due to merge conflicts - apologies.


Changes in v2
=============

 * added .gitattributes file to enforce binary diffs for the test refs that
   cannot be applied when being sent via e-mail
 * perform filter graph re-init due to subtitle "frame size" change only
   when the size was unknown before and not set via -canvas_size
 * overlaytextsubs: Make sure to request frames on the subtitle input
 * avfilter/splitcc: Start parsing cc data on key frames only
 * avcodec/webvttenc: Don't encode ass drawing codes and empty lines
 * stripstyles: fix mem leak
 * gs2t: improve color detection
 * gs2t: empty frames must not be skipped
 * subfeed: fix name
 * textmod: preserve margins
 * added .gitattributes file to enforce binary diffs for the test refs that
   cannot be applied when being sent via e-mail
 * perform filter graph re-init due to subtitle "frame size" change only
   when the size was unknown before and not set via -canvas_size
 * avcodec/dvbsubdec: Fix conditions for fallback to default resolution
 * Made changes suggested by Andreas
 * Fixed failing command line reported by Michael

Changes from previous version v24:


AVFrame
=======

 * Removed sub_start_time The start time is now added to the subtitle
   start_pts during decoding The sub_end_time field is adjusted accordingly
 * Renamed sub_end_time to duration which it is effectively after removing
   the start_time
 * Added a sub-struct 'subtitle_timing' to av frame Contains subtitle_pts
   renamed to 'subtitle_timing.start_pts' and 'subtitle_timing.duration'
 * Change both fields to (fixed) time_base AV_TIMEBASE
 * add repeat_sub field provides a clear indication whether a subtitle frame
   is an actual subtitle event or a repeated subtitle frame in a filter
   graph


Heartbeat Removal
=================

 * completely removed the earlier heartbeat implementation
 * filtering arbitration is now implemented in a new filter: 'subfeed'
 * subfeed will be auto-inserted for compatiblity with sub2video command
   lines
 * the new behavior is not exactly identical to the earlier behavior, but it
   basically allows to achieve the same results
 * there's a small remainder, now named subtitle kickoff which serves to get
   things (in the filter graph) going right from the start


New 'subfeed' Filter
====================

 * a versatile filter for solving all kinds of problems with subtile frame
   flow in filter graphs
 * Can be inserted at any position in a graph
 * Auto-inserted for sub2video command lines (in repeat-mode)
 * Allows duration fixup delay input frames with unknown duration and infer
   duration from start of subsequent frame
 * Provides multiple modes of operation:
   * repeat mode (default) Queues input frames Outputs frames at a fixed
     (configurable) rate Either sends a matching input frame (repeatedly) or
     empty frames otherwise
   * scatter mode similar to repeat mode, but splits input frames by
     duration into small segments with same content
   * forward mode No fixed output rate Useful in combination with duration
     fixup or overlap fixup


ffmpeg Tool Changes
===================

 * delay subtitle output stream initialization (like for audio and video)
   This is needed for example when a format header depends on having
   received an initial frame to derive certain header values from
 * decoding: set subtitle frame size from decoding context
 * re-init graph when subtitle size changes
 * always insert subscale filter for sub2video command lines (to ensure
   correct scaling)


Subtitle Encoding
=================

 * ignore repeated frames for encoding based on repeat_sub field in AVFrame
 * support multi-area encoding for text subtitles Subtitle OCR can create
   multiple areas at different positions. Previously, the texts were always
   squashed into a single area ('subtitle rect'), which was not ideal.
   Multiple text areas are now generally supported:
   * ASS Encoder Changed to use the 'receive_packet' encoding API A single
     frame with multiple text areas will create multiple packets now
   * All other text subtitle encoders A newline is inserted between the text
     from multiple areas


graphicsub2text (OCR)
=====================

 * enhanced preprocessing
   * using elbg algorithm for color quantization
   * detection and removal of text outlines
   * map-based identification of colors per word (text, outline, background)
 * add option for duration fixup
 * add option to dump preprocessing bitmaps
 * Recognize formatting and apply as ASS inline styles
   * per word(!)
   * paragraph alignment
   * positioning
   * font names
   * font size
   * font style (italic, underline, bold)
   * text color, outline color


Other Filter Changes
====================

 * all: Make sure to forward all link properties (time base, frame rate, w,
   h) where appropriate
 * overlaytextsubs: request frames on the subtitle input
 * overlaytextsubs: disable read-order checking
 * overlaytextsubs: improve implementation of render_latest_only
 * overlaytextsubs: ensure equal in/out video formats
 * splitcc: derive framerate from realtime_latency
 * graphicsub2video: implement caching of converted frames
 * graphicsub2video: use 1x1 output frame size as long as subtitle size is
   unknown (0x0)

Plus a dozen of things I forgot..

softworkz (23):
  avcodec,avutil: Move enum AVSubtitleType to avutil, add new and
    deprecate old values
  avutil/frame: Prepare AVFrame for subtitle handling
  avcodec/subtitles: Introduce new frame-based subtitle decoding API
  avcodec/libzvbi: set subtitle type
  avfilter/subtitles: Update vf_subtitles to use new decoding api
  avcodec,avutil: Move ass helper functions to avutil as avpriv_ and
    extend ass dialog parsing
  avcodec/subtitles: Replace deprecated enum values
  fftools/play,probe: Adjust for subtitle changes
  avfilter/subtitles: Add subtitles.c for subtitle frame allocation
  avfilter/avfilter: Handle subtitle frames
  avfilter/avfilter: Fix hardcoded input index
  avfilter/sbuffer: Add sbuffersrc and sbuffersink filters
  avfilter/overlaygraphicsubs: Add overlaygraphicsubs and
    graphicsub2video filters
  avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video
    filters
  avfilter/textmod: Add textmod, censor and show_speaker filters
  avfilter/stripstyles: Add stripstyles filter
  avfilter/splitcc: Add splitcc filter for closed caption handling
  avfilter/graphicsub2text: Add new graphicsub2text filter (OCR)
  avfilter/subscale: Add filter for scaling and/or re-arranging
    graphical subtitles
  avfilter/subfeed: add subtitle feed filter
  avcodec/subtitles: Migrate subtitle encoders to frame-based API
  fftools/ffmpeg: Introduce subtitle filtering and new frame-based
    subtitle encoding
  avcodec/dvbsubdec: Fix conditions for fallback to default resolution

 configure                                     |    7 +-
 doc/filters.texi                              |  756 ++++++++++
 fftools/ffmpeg.c                              |  610 ++++----
 fftools/ffmpeg.h                              |   17 +-
 fftools/ffmpeg_filter.c                       |  243 +++-
 fftools/ffmpeg_hw.c                           |    2 +-
 fftools/ffmpeg_opt.c                          |    4 +-
 fftools/ffplay.c                              |  102 +-
 fftools/ffprobe.c                             |   47 +-
 libavcodec/Makefile                           |   56 +-
 libavcodec/ass.h                              |  151 +-
 libavcodec/assdec.c                           |    4 +-
 libavcodec/assenc.c                           |  191 ++-
 libavcodec/avcodec.c                          |    8 +
 libavcodec/avcodec.h                          |   34 +-
 libavcodec/ccaption_dec.c                     |   20 +-
 libavcodec/codec_internal.h                   |   12 -
 libavcodec/decode.c                           |   60 +-
 libavcodec/dvbsubdec.c                        |   53 +-
 libavcodec/dvbsubenc.c                        |   96 +-
 libavcodec/dvdsubdec.c                        |    2 +-
 libavcodec/dvdsubenc.c                        |  102 +-
 libavcodec/encode.c                           |   61 +-
 libavcodec/internal.h                         |   16 +
 libavcodec/jacosubdec.c                       |    2 +-
 libavcodec/libaribb24.c                       |    2 +-
 libavcodec/libzvbi-teletextdec.c              |   17 +-
 libavcodec/microdvddec.c                      |    7 +-
 libavcodec/movtextdec.c                       |    3 +-
 libavcodec/movtextenc.c                       |  126 +-
 libavcodec/mpl2dec.c                          |    2 +-
 libavcodec/pgssubdec.c                        |    2 +-
 libavcodec/realtextdec.c                      |    2 +-
 libavcodec/samidec.c                          |    2 +-
 libavcodec/srtdec.c                           |    2 +-
 libavcodec/srtenc.c                           |  116 +-
 libavcodec/subviewerdec.c                     |    2 +-
 libavcodec/tests/avcodec.c                    |    5 +-
 libavcodec/textdec.c                          |    4 +-
 libavcodec/ttmlenc.c                          |  114 +-
 libavcodec/utils.c                            |  185 ++-
 libavcodec/webvttdec.c                        |    2 +-
 libavcodec/webvttenc.c                        |   94 +-
 libavcodec/xsubdec.c                          |    2 +-
 libavcodec/xsubenc.c                          |   88 +-
 libavfilter/Makefile                          |   15 +
 libavfilter/allfilters.c                      |   14 +
 libavfilter/avfilter.c                        |   34 +-
 libavfilter/avfilter.h                        |   11 +
 libavfilter/avfiltergraph.c                   |    5 +
 libavfilter/buffersink.c                      |   54 +
 libavfilter/buffersink.h                      |    7 +
 libavfilter/buffersrc.c                       |   72 +
 libavfilter/buffersrc.h                       |    1 +
 libavfilter/formats.c                         |   16 +
 libavfilter/formats.h                         |    3 +
 libavfilter/internal.h                        |   19 +-
 libavfilter/sf_graphicsub2text.c              | 1132 +++++++++++++++
 libavfilter/sf_splitcc.c                      |  395 ++++++
 libavfilter/sf_stripstyles.c                  |  211 +++
 libavfilter/sf_subfeed.c                      |  402 ++++++
 libavfilter/sf_subscale.c                     |  884 ++++++++++++
 libavfilter/sf_textmod.c                      |  710 ++++++++++
 libavfilter/subtitles.c                       |   63 +
 libavfilter/subtitles.h                       |   44 +
 libavfilter/vf_overlaygraphicsubs.c           |  765 ++++++++++
 libavfilter/vf_overlaytextsubs.c              |  680 +++++++++
 libavfilter/vf_subtitles.c                    |   67 +-
 libavutil/Makefile                            |    4 +
 {libavcodec => libavutil}/ass.c               |  115 +-
 libavutil/ass_internal.h                      |  135 ++
 {libavcodec => libavutil}/ass_split.c         |   30 +-
 .../ass_split_internal.h                      |   32 +-
 libavutil/frame.c                             |  211 ++-
 libavutil/frame.h                             |   85 +-
 libavutil/subfmt.c                            |   45 +
 libavutil/subfmt.h                            |  115 ++
 libavutil/version.h                           |    1 +
 tests/ref/fate/filter-overlay-dvdsub-2397     |  182 +--
 tests/ref/fate/sub-dvb                        |  162 ++-
 tests/ref/fate/sub2video                      | 1091 ++++++++++++++-
 tests/ref/fate/sub2video_basic                | 1238 +++++++++++++++--
 tests/ref/fate/sub2video_time_limited         |   78 +-
 83 files changed, 11149 insertions(+), 1412 deletions(-)
 create mode 100644 libavfilter/sf_graphicsub2text.c
 create mode 100644 libavfilter/sf_splitcc.c
 create mode 100644 libavfilter/sf_stripstyles.c
 create mode 100644 libavfilter/sf_subfeed.c
 create mode 100644 libavfilter/sf_subscale.c
 create mode 100644 libavfilter/sf_textmod.c
 create mode 100644 libavfilter/subtitles.c
 create mode 100644 libavfilter/subtitles.h
 create mode 100644 libavfilter/vf_overlaygraphicsubs.c
 create mode 100644 libavfilter/vf_overlaytextsubs.c
 rename {libavcodec => libavutil}/ass.c (59%)
 create mode 100644 libavutil/ass_internal.h
 rename {libavcodec => libavutil}/ass_split.c (94%)
 rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%)
 create mode 100644 libavutil/subfmt.c
 create mode 100644 libavutil/subfmt.h


base-commit: 9fba0b8a8c754a012fc74c90ffb7c26a56be8ca0
Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-18%2Fsoftworkz%2Fsubmit_subfiltering-v4
Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-18/softworkz/submit_subfiltering-v4
Pull-Request: https://github.com/ffstaging/FFmpeg/pull/18

Range-diff vs v3:

  1:  7767933235 !  1:  2f3ba171f5 avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values
     @@ libavutil/subfmt.h (new)
      
       ## libavutil/version.h ##
      @@
     - #define FF_API_COLORSPACE_NAME          (LIBAVUTIL_VERSION_MAJOR < 58)
     - #define FF_API_AV_MALLOCZ_ARRAY         (LIBAVUTIL_VERSION_MAJOR < 58)
     - #define FF_API_FIFO_PEEK2               (LIBAVUTIL_VERSION_MAJOR < 58)
     + #define FF_API_XVMC                     (LIBAVUTIL_VERSION_MAJOR < 58)
     + #define FF_API_OLD_CHANNEL_LAYOUT       (LIBAVUTIL_VERSION_MAJOR < 58)
     + #define FF_API_AV_FOPEN_UTF8            (LIBAVUTIL_VERSION_MAJOR < 58)
      +#define FF_API_OLD_SUBTITLES            (LIBAVUTIL_VERSION_MAJOR < 58)
       
       /**
  2:  e922f141ba !  2:  ff101f8a76 avutil/frame: Prepare AVFrame for subtitle handling
     @@ libavutil/frame.c
      +#include "subfmt.h"
       #include "hwcontext.h"
       
     - #define CHECK_CHANNELS_CONSISTENCY(frame) \
     + #if FF_API_OLD_CHANNEL_LAYOUT
      @@ libavutil/frame.c: const char *av_get_colorspace_name(enum AVColorSpace val)
           return name[val];
       }
     @@ libavutil/frame.c: static void get_frame_defaults(AVFrame *frame)
       }
       
       static void free_side_data(AVFrameSideData **ptr_sd)
     -@@ libavutil/frame.c: static int get_audio_buffer(AVFrame *frame, int align)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
       
       }
       
     @@ libavutil/frame.c: static int get_audio_buffer(AVFrame *frame, int align)
      +}
      +
       int av_frame_get_buffer(AVFrame *frame, int align)
     -+{
     -+    if (frame->width > 0 && frame->height > 0)
     + {
     +     if (frame->format < 0)
     +@@ libavutil/frame.c: int av_frame_get_buffer(AVFrame *frame, int align)
     + 
     + FF_DISABLE_DEPRECATION_WARNINGS
     +     if (frame->width > 0 && frame->height > 0)
     +-        return get_video_buffer(frame, align);
      +        frame->type = AVMEDIA_TYPE_VIDEO;
     -+    else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0))
     +     else if (frame->nb_samples > 0 &&
     +              (av_channel_layout_check(&frame->ch_layout)
     + #if FF_API_OLD_CHANNEL_LAYOUT
     +               || frame->channel_layout || frame->channels > 0
     + #endif
     +              ))
     +-        return get_audio_buffer(frame, align);
      +        frame->type = AVMEDIA_TYPE_AUDIO;
     -+
     + FF_ENABLE_DEPRECATION_WARNINGS
     + 
     +-    return AVERROR(EINVAL);
      +    return av_frame_get_buffer2(frame, align);
      +}
      +
      +int av_frame_get_buffer2(AVFrame *frame, int align)
     - {
     -     if (frame->format < 0)
     -         return AVERROR(EINVAL);
     - 
     --    if (frame->width > 0 && frame->height > 0)
     -+    switch(frame->type) {
     ++{
     ++    if (frame->format < 0)
     ++        return AVERROR(EINVAL);
     ++
     ++    switch (frame->type) {
      +    case AVMEDIA_TYPE_VIDEO:
     -         return get_video_buffer(frame, align);
     --    else if (frame->nb_samples > 0 && (frame->channel_layout || frame->channels > 0))
     ++        return get_video_buffer(frame, align);
      +    case AVMEDIA_TYPE_AUDIO:
     -         return get_audio_buffer(frame, align);
     --
     --    return AVERROR(EINVAL);
     ++        return get_audio_buffer(frame, align);
      +    case AVMEDIA_TYPE_SUBTITLE:
      +        return get_subtitle_buffer(frame);
      +    default:
     @@ libavutil/frame.c: static int frame_copy_props(AVFrame *dst, const AVFrame *src,
       
           av_dict_copy(&dst->metadata, src->metadata, 0);
       
     -@@ libavutil/frame.c: int av_frame_ref(AVFrame *dst, const AVFrame *src)
     -     av_assert1(dst->width == 0 && dst->height == 0);
     -     av_assert1(dst->channels == 0);
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
     +     av_assert1(dst->ch_layout.nb_channels == 0 &&
     +                dst->ch_layout.order == AV_CHANNEL_ORDER_UNSPEC);
       
      +    dst->type           = src->type;
           dst->format         = src->format;
           dst->width          = src->width;
           dst->height         = src->height;
     -@@ libavutil/frame.c: int av_frame_ref(AVFrame *dst, const AVFrame *src)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
       
           /* duplicate the frame data if it's not refcounted */
           if (!src->buf[0]) {
     @@ libavutil/frame.c: int av_frame_ref(AVFrame *dst, const AVFrame *src)
               if (ret < 0)
                   goto fail;
       
     -@@ libavutil/frame.c: int av_frame_ref(AVFrame *dst, const AVFrame *src)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
               }
           }
       
     @@ libavutil/frame.c: void av_frame_unref(AVFrame *frame)
           if (frame->extended_data != frame->data)
               av_freep(&frame->extended_data);
       
     -@@ libavutil/frame.c: void av_frame_move_ref(AVFrame *dst, AVFrame *src)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
       
       int av_frame_is_writable(AVFrame *frame)
       {
     @@ libavutil/frame.c: int av_frame_make_writable(AVFrame *frame)
           tmp.format         = frame->format;
           tmp.width          = frame->width;
           tmp.height         = frame->height;
     -@@ libavutil/frame.c: int av_frame_make_writable(AVFrame *frame)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
           if (frame->hw_frames_ctx)
               ret = av_hwframe_get_buffer(frame->hw_frames_ctx, &tmp, 0);
           else
     @@ libavutil/frame.c: AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int pl
           int planes, i;
       
      -    if (frame->nb_samples) {
     --        int channels = frame->channels;
     --        if (!channels)
     --            return NULL;
     --        CHECK_CHANNELS_CONSISTENCY(frame);
     --        planes = av_sample_fmt_is_planar(frame->format) ? channels : 1;
     --    } else
      +    switch(frame->type) {
      +    case AVMEDIA_TYPE_VIDEO:
     -         planes = 4;
     ++        planes = 4;
      +        break;
      +    case AVMEDIA_TYPE_AUDIO:
      +        {
     -+            int channels = frame->channels;
     -+            if (!channels)
     -+                return NULL;
     -+            CHECK_CHANNELS_CONSISTENCY(frame);
     -+            planes = av_sample_fmt_is_planar(frame->format) ? channels : 1;
     -+            break;
     +         int channels = frame->ch_layout.nb_channels;
     + 
     + #if FF_API_OLD_CHANNEL_LAYOUT
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
     +         if (!channels)
     +             return NULL;
     +         planes = av_sample_fmt_is_planar(frame->format) ? channels : 1;
     +-    } else
     +-        planes = 4;
     ++        break;
      +        }
      +    default:
      +        return NULL;
     @@ libavutil/frame.c: AVBufferRef *av_frame_get_plane_buffer(AVFrame *frame, int pl
       
           if (plane < 0 || plane >= planes || !frame->extended_data[plane])
               return NULL;
     -@@ libavutil/frame.c: static int frame_copy_audio(AVFrame *dst, const AVFrame *src)
     +@@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
           return 0;
       }
       
     @@ libavutil/frame.c: static int frame_copy_audio(AVFrame *dst, const AVFrame *src)
           if (dst->format != src->format || dst->format < 0)
               return AVERROR(EINVAL);
       
     +-FF_DISABLE_DEPRECATION_WARNINGS
      -    if (dst->width > 0 && dst->height > 0)
      +    switch(dst->type) {
      +    case AVMEDIA_TYPE_VIDEO:
               return frame_copy_video(dst, src);
     --    else if (dst->nb_samples > 0 && dst->channels > 0)
     +-    else if (dst->nb_samples > 0 &&
     +-             (av_channel_layout_check(&dst->ch_layout)
     +-#if FF_API_OLD_CHANNEL_LAYOUT
     +-              || dst->channel_layout || dst->channels
     +-#endif
     +-            ))
      +    case AVMEDIA_TYPE_AUDIO:
               return frame_copy_audio(dst, src);
     +-FF_ENABLE_DEPRECATION_WARNINGS
      -
      -    return AVERROR(EINVAL);
      +    case AVMEDIA_TYPE_SUBTITLE:
     @@ libavutil/frame.h: typedef struct AVFrame {
           int format;
       
      @@ libavutil/frame.h: typedef struct AVFrame {
     -      * for the target frame's private_ref field.
     +      * Channel layout of the audio data.
            */
     -     AVBufferRef *private_ref;
     +     AVChannelLayout ch_layout;
      +
      +    /**
      +     * Media type of the frame (audio, video, subtitles..)
  3:  ec262914b0 !  3:  b8935d5e68 avcodec/subtitles: Introduce new frame-based subtitle decoding API
     @@ Commit message
      
          Signed-off-by: softworkz <softworkz@hotmail.com>
      
     + ## libavcodec/avcodec.c ##
     +@@ libavcodec/avcodec.c: FF_DISABLE_DEPRECATION_WARNINGS
     + FF_ENABLE_DEPRECATION_WARNINGS
     + #endif
     + 
     ++        // Set the subtitle type from the codec descriptor in case the decoder hasn't done itself
     ++        if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE && avctx->subtitle_type == AV_SUBTITLE_FMT_UNKNOWN) {
     ++            if(avctx->codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB)
     ++                avctx->subtitle_type = AV_SUBTITLE_FMT_BITMAP;
     ++            if(avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB)
     ++                 avctx->subtitle_type = AV_SUBTITLE_FMT_ASS;
     ++        }
     ++
     + #if FF_API_AVCTX_TIMEBASE
     +         if (avctx->framerate.num > 0 && avctx->framerate.den > 0)
     +             avctx->time_base = av_inv_q(av_mul_q(avctx->framerate, (AVRational){avctx->ticks_per_frame, 1}));
     +
       ## libavcodec/avcodec.h ##
      @@ libavcodec/avcodec.h: typedef struct AVCodecContext {
       
     @@ libavcodec/avcodec.h: typedef struct AVCodecContext {
            * [Script Info] and [V4+ Styles] section, plus the [Events] line and
            * the Format line following. It shouldn't include any Dialogue line.
            * - encoding: Set/allocated/freed by user (before avcodec_open2())
     +@@ libavcodec/avcodec.h: typedef struct AVCodecContext {
     +      *             The decoder can then override during decoding as needed.
     +      */
     +     AVChannelLayout ch_layout;
     ++
     ++    enum AVSubtitleType subtitle_type;
     + } AVCodecContext;
     + 
     + /**
      @@ libavcodec/avcodec.h: int avcodec_close(AVCodecContext *avctx);
        * Free all allocated data in the given subtitle struct.
        *
     @@ libavcodec/avcodec.h: enum AVChromaLocation avcodec_chroma_pos_to_enum(int xpos,
                                   int *got_sub_ptr,
                                   AVPacket *avpkt);
      
     - ## libavcodec/codec_desc.c ##
     -@@ libavcodec/codec_desc.c: enum AVMediaType avcodec_get_type(enum AVCodecID codec_id)
     -     const AVCodecDescriptor *desc = avcodec_descriptor_get(codec_id);
     -     return desc ? desc->type : AVMEDIA_TYPE_UNKNOWN;
     - }
     -+
     -+enum AVSubtitleType avcodec_descriptor_get_subtitle_format(const AVCodecDescriptor *codec_descriptor)
     -+{
     -+    if(codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB)
     -+        return AV_SUBTITLE_FMT_BITMAP;
     -+
     -+    if(codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB)
     -+        return AV_SUBTITLE_FMT_ASS;
     -+
     -+    return AV_SUBTITLE_FMT_UNKNOWN;
     -+}
     -
     - ## libavcodec/codec_desc.h ##
     -@@ libavcodec/codec_desc.h: const AVCodecDescriptor *avcodec_descriptor_next(const AVCodecDescriptor *prev);
     -  */
     - const AVCodecDescriptor *avcodec_descriptor_get_by_name(const char *name);
     - 
     -+/**
     -+ * Return subtitle format from a codec descriptor
     -+ *
     -+ * @param codec_descriptor codec descriptor
     -+ * @return                 the subtitle type (e.g. bitmap, text)
     -+ */
     -+enum AVSubtitleType avcodec_descriptor_get_subtitle_format(const AVCodecDescriptor *codec_descriptor);
     -+
     - /**
     -  * @}
     -  */
     -
       ## libavcodec/decode.c ##
      @@ libavcodec/decode.c: static int decode_receive_frame_internal(AVCodecContext *avctx, AVFrame *frame)
           return ret;
     @@ libavcodec/decode.c: int attribute_align_arg avcodec_send_packet(AVCodecContext
               return AVERROR(EINVAL);
       
      +    if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE)
     -+		// this does not exactly implement the avcodec_send_packet/avcodec_receive_frame API 
     ++		// this does not exactly implement the avcodec_send_packet/avcodec_receive_frame API
      +	    // but we know that no subtitle decoder produces multiple AVSubtitles per packet through
      +		// the legacy API, and this will be changed when migrating the subtitle decoders
      +		// to the frame based decoding api
     @@ libavcodec/decode.c: int avcodec_decode_subtitle2(AVCodecContext *avctx, AVSubti
      -            sub->format = 0;
      -        else if (avctx->codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB)
      -            sub->format = 1;
     -+        sub->format = avcodec_descriptor_get_subtitle_format(avctx->codec_descriptor);
     ++        sub->format = (uint16_t)avctx->subtitle_type;
       
               for (unsigned i = 0; i < sub->num_rects; i++) {
                   if (avctx->sub_charenc_mode != FF_SUB_CHARENC_MODE_IGNORE &&
     @@ libavcodec/internal.h: int ff_int_from_list_or_default(void *ctx, const char * v
       #endif /* AVCODEC_INTERNAL_H */
      
       ## libavcodec/utils.c ##
     -@@ libavcodec/utils.c: int av_get_audio_frame_duration(AVCodecContext *avctx, int frame_bytes)
     +@@ libavcodec/utils.c: FF_ENABLE_DEPRECATION_WARNINGS
           return FFMAX(0, duration);
       }
       
     @@ libavcodec/utils.c: int av_get_audio_frame_duration(AVCodecContext *avctx, int f
      +
       int av_get_audio_frame_duration2(AVCodecParameters *par, int frame_bytes)
       {
     -     int duration = get_audio_frame_duration(par->codec_id, par->sample_rate,
     +    int channels = par->ch_layout.nb_channels;
  -:  ---------- >  4:  4b44732e07 avcodec/libzvbi: set subtitle type
  4:  77bd67ee37 !  5:  8faa7a4043 avfilter/subtitles: Update vf_subtitles to use new decoding api
     @@ libavfilter/vf_subtitles.c: static int attachment_is_font(AVStream * st)
      +
       AVFILTER_DEFINE_CLASS(subtitles);
       
     ++static enum AVSubtitleType get_subtitle_format(const AVCodecDescriptor *codec_descriptor)
     ++{
     ++    if(codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB)
     ++        return AV_SUBTITLE_FMT_BITMAP;
     ++
     ++    if(codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB)
     ++        return AV_SUBTITLE_FMT_ASS;
     ++
     ++    return AV_SUBTITLE_FMT_UNKNOWN;
     ++}
     ++
       static av_cold int init_subtitles(AVFilterContext *ctx)
     + {
     +     int j, ret, sid;
      @@ libavfilter/vf_subtitles.c: static av_cold int init_subtitles(AVFilterContext *ctx)
           AVStream *st;
           AVPacket pkt;
     @@ libavfilter/vf_subtitles.c: static av_cold int init_subtitles(AVFilterContext *c
      +
           dec_desc = avcodec_descriptor_get(st->codecpar->codec_id);
      -    if (dec_desc && !(dec_desc->props & AV_CODEC_PROP_TEXT_SUB)) {
     -+    subtitle_format = avcodec_descriptor_get_subtitle_format(dec_desc);
     ++    subtitle_format = get_subtitle_format(dec_desc);
      +
      +    if (subtitle_format != AV_SUBTITLE_FMT_ASS) {
               av_log(ctx, AV_LOG_ERROR,
  5:  26bad0c088 !  6:  1664026d7c avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing
     @@ libavcodec/ass.h
      +static inline int ff_ass_subtitle_header_default(AVCodecContext *avctx)
      +{
      +    avctx->subtitle_header = (uint8_t *)avpriv_ass_get_subtitle_header_default(!(avctx->flags & AV_CODEC_FLAG_BITEXACT));
     -+
     + 
     +-/**
     +- * Add an ASS dialog to a subtitle.
     +- */
     +-int ff_ass_add_rect(AVSubtitle *sub, const char *dialog,
     +-                    int readorder, int layer, const char *style,
     +-                    const char *speaker);
      +    if (!avctx->subtitle_header)
      +        return AVERROR(ENOMEM);
      +    avctx->subtitle_header_size = strlen((char *)avctx->subtitle_header);
     @@ libavcodec/ass.h
       /**
        * Add an ASS dialog to a subtitle.
        */
     --int ff_ass_add_rect(AVSubtitle *sub, const char *dialog,
     +-int ff_ass_add_rect2(AVSubtitle *sub, const char *dialog,
     +-                     int readorder, int layer, const char *style,
     +-                     const char *speaker, unsigned *nb_rect_allocated);
      +static inline int avpriv_ass_add_rect(AVSubtitle *sub, const char *dialog,
     -                     int readorder, int layer, const char *style,
     --                    const char *speaker);
     ++                    int readorder, int layer, const char *style,
      +                    const char *speaker)
      +{
      +    char *ass_str;
     @@ libavcodec/assdec.c
       
       #include "avcodec.h"
      -#include "ass.h"
     + #include "codec_internal.h"
     + #include "config_components.h"
      +#include "libavutil/ass_internal.h"
     - #include "internal.h"
       #include "libavutil/internal.h"
       #include "libavutil/mem.h"
     + 
      
       ## libavcodec/assenc.c ##
      @@
     @@ libavcodec/assenc.c
       
       #include "avcodec.h"
      -#include "ass.h"
     + #include "codec_internal.h"
      +#include "libavutil/ass_internal.h"
     - #include "internal.h"
       #include "libavutil/avstring.h"
       #include "libavutil/internal.h"
     + #include "libavutil/mem.h"
      
       ## libavcodec/ccaption_dec.c ##
      @@ libavcodec/ccaption_dec.c: static av_cold int init_decoder(AVCodecContext *avctx)
     @@ libavcodec/ccaption_dec.c: static av_cold int init_decoder(AVCodecContext *avctx
           if (ret < 0) {
               return ret;
           }
     -@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, void *data, int *got_sub, AVPacket *avp
     +@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, AVSubtitle *sub,
     +     int len = avpkt->size;
     +     int ret = 0;
     +     int i;
     +-    unsigned nb_rect_allocated = 0;
     + 
     +     for (i = 0; i < len; i += 3) {
     +         uint8_t hi, cc_type = bptr[i] & 1;
     +@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, AVSubtitle *sub,
                                                            AV_TIME_BASE_Q, ms_tb);
                   else
                       sub->end_display_time = -1;
     --            ret = ff_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
     +-            ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated);
      +            ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
                   if (ret < 0)
                       return ret;
                   ctx->last_real_time = sub->pts;
     -@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, void *data, int *got_sub, AVPacket *avp
     +@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, AVSubtitle *sub,
       
           if (!bptr && !ctx->real_time && ctx->buffer[!ctx->buffer_index].str[0]) {
               bidx = !ctx->buffer_index;
     --        ret = ff_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
     +-        ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated);
      +        ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
               if (ret < 0)
                   return ret;
               sub->pts = ctx->buffer_time[1];
     -@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, void *data, int *got_sub, AVPacket *avp
     +@@ libavcodec/ccaption_dec.c: static int decode(AVCodecContext *avctx, AVSubtitle *sub,
               capture_screen(ctx);
               ctx->buffer_changed = 0;
       
     --        ret = ff_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
     +-        ret = ff_ass_add_rect2(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL, &nb_rect_allocated);
      +        ret = avpriv_ass_add_rect(sub, ctx->buffer[bidx].str, ctx->readorder++, 0, NULL, NULL);
               if (ret < 0)
                   return ret;
               sub->end_display_time = -1;
      
       ## libavcodec/jacosubdec.c ##
     -@@ libavcodec/jacosubdec.c: static int jacosub_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/jacosubdec.c: static int jacosub_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
               av_bprint_init(&buffer, JSS_MAX_LINESIZE, JSS_MAX_LINESIZE);
               jacosub_to_ass(avctx, &buffer, ptr);
     @@ libavcodec/libzvbi-teletextdec.c: static int gen_sub_bitmap(TeletextContext *ctx
               return 0;
           }
       
     -@@ libavcodec/libzvbi-teletextdec.c: static int teletext_decode_frame(AVCodecContext *avctx, void *data, int *got_sub
     +@@ libavcodec/libzvbi-teletextdec.c: static int teletext_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
               sub->num_rects = 0;
               sub->pts = ctx->pages->pts;
       
     @@ libavcodec/libzvbi-teletextdec.c: static int teletext_decode_frame(AVCodecContex
                       sub->num_rects = 1;
      
       ## libavcodec/microdvddec.c ##
     -@@ libavcodec/microdvddec.c: static int microdvd_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/microdvddec.c: static int microdvd_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
               }
           }
           if (new_line.len) {
     @@ libavcodec/movtextdec.c
       #include "libavutil/common.h"
       #include "libavutil/bprint.h"
       #include "libavutil/intreadwrite.h"
     -@@ libavcodec/movtextdec.c: static int mov_text_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/movtextdec.c: static int mov_text_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
           } else
               text_to_ass(&buf, ptr, end, avctx);
       
     @@ libavcodec/movtextenc.c
      +#include "libavutil/ass_split_internal.h"
      +#include "libavutil/ass_internal.h"
       #include "bytestream.h"
     - #include "internal.h"
     + #include "codec_internal.h"
       
      @@ libavcodec/movtextenc.c: static int mov_text_encode_close(AVCodecContext *avctx)
       {
     @@ libavcodec/movtextenc.c: static int mov_text_encode_frame(AVCodecContext *avctx,
           if (s->buffer.len > UINT16_MAX)
      
       ## libavcodec/mpl2dec.c ##
     -@@ libavcodec/mpl2dec.c: static int mpl2_decode_frame(AVCodecContext *avctx, void *data,
     +@@ libavcodec/mpl2dec.c: static int mpl2_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED);
           if (ptr && avpkt->size > 0 && *ptr && !mpl2_event_to_ass(&buf, ptr))
     @@ libavcodec/mpl2dec.c: static int mpl2_decode_frame(AVCodecContext *avctx, void *
               return ret;
      
       ## libavcodec/realtextdec.c ##
     -@@ libavcodec/realtextdec.c: static int realtext_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/realtextdec.c: static int realtext_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           av_bprint_init(&buf, 0, 4096);
           if (ptr && avpkt->size > 0 && !rt_event_to_ass(&buf, ptr))
     @@ libavcodec/realtextdec.c: static int realtext_decode_frame(AVCodecContext *avctx
               return ret;
      
       ## libavcodec/samidec.c ##
     -@@ libavcodec/samidec.c: static int sami_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/samidec.c: static int sami_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
               if (ret < 0)
                   return ret;
               // TODO: pass escaped sami->encoded_source.str as source
     @@ libavcodec/samidec.c: static int sami_decode_frame(AVCodecContext *avctx,
           }
      
       ## libavcodec/srtdec.c ##
     -@@ libavcodec/srtdec.c: static int srt_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/srtdec.c: static int srt_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           ret = srt_to_ass(avctx, &buffer, avpkt->data, x1, y1, x2, y2);
           if (ret >= 0)
     @@ libavcodec/srtenc.c
       #include "libavutil/bprint.h"
      -#include "ass_split.h"
      -#include "ass.h"
     + #include "codec_internal.h"
      +#include "libavutil/ass_split_internal.h"
      +#include "libavutil/ass_internal.h"
     - #include "internal.h"
       
       
     + #define SRT_STACK_SIZE 64
      @@ libavcodec/srtenc.c: static void srt_stack_push_pop(SRTContext *s, const char c, int close)
       
       static void srt_style_apply(SRTContext *s, const char *style)
     @@ libavcodec/srtenc.c: static int text_encode_frame(AVCodecContext *avctx,
       }
      
       ## libavcodec/subviewerdec.c ##
     -@@ libavcodec/subviewerdec.c: static int subviewer_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/subviewerdec.c: static int subviewer_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED);
           if (ptr && avpkt->size > 0 && !subviewer_event_to_ass(&buf, ptr))
     @@ libavcodec/subviewerdec.c: static int subviewer_decode_frame(AVCodecContext *avc
               return ret;
      
       ## libavcodec/textdec.c ##
     -@@ libavcodec/textdec.c: static int text_decode_frame(AVCodecContext *avctx, void *data,
     +@@ libavcodec/textdec.c: static int text_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED);
           if (ptr && avpkt->size > 0 && *ptr) {
     @@ libavcodec/ttmlenc.c: static av_cold int ttml_encode_init(AVCodecContext *avctx)
       
      
       ## libavcodec/webvttdec.c ##
     -@@ libavcodec/webvttdec.c: static int webvtt_decode_frame(AVCodecContext *avctx,
     +@@ libavcodec/webvttdec.c: static int webvtt_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
       
           av_bprint_init(&buf, 0, AV_BPRINT_SIZE_UNLIMITED);
           if (ptr && avpkt->size > 0 && !webvtt_event_to_ass(&buf, ptr))
     @@ libavcodec/webvttenc.c
       #include "libavutil/bprint.h"
      -#include "ass_split.h"
      -#include "ass.h"
     + #include "codec_internal.h"
      +#include "libavutil/ass_split_internal.h"
      +#include "libavutil/ass_internal.h"
     - #include "internal.h"
       
       #define WEBVTT_STACK_SIZE 64
     + typedef struct {
      @@ libavcodec/webvttenc.c: static void webvtt_stack_push_pop(WebVTTContext *s, const char c, int close)
       
       static void webvtt_style_apply(WebVTTContext *s, const char *style)
     @@ libavutil/ass.c: char *ff_ass_get_dialog(int readorder, int layer, const char *s
                              speaker ? speaker : "", text);
       }
       
     --int ff_ass_add_rect(AVSubtitle *sub, const char *dialog,
     +-int ff_ass_add_rect2(AVSubtitle *sub, const char *dialog,
      -                    int readorder, int layer, const char *style,
     --                    const char *speaker)
     -+char *avpriv_ass_get_dialog_ex(int readorder, int layer, const char *style,
     -+                        const char *speaker, int margin_l, int margin_r,
     -+                        int margin_v, const char *text)
     - {
     --    AVSubtitleRect **rects, *rect;
     +-                    const char *speaker, unsigned *nb_rect_allocated)
     +-{
     +-    AVSubtitleRect **rects = sub->rects, *rect;
      -    char *ass_str;
     +-    uint64_t new_nb = 0;
      -
     --    rects = av_realloc_array(sub->rects, sub->num_rects+1, sizeof(*sub->rects));
     --    if (!rects)
     +-    if (sub->num_rects >= UINT_MAX)
      -        return AVERROR(ENOMEM);
     --    sub->rects = rects;
     +-
     +-    if (nb_rect_allocated && *nb_rect_allocated <= sub->num_rects) {
     +-        if (sub->num_rects < UINT_MAX / 17 * 16) {
     +-            new_nb = sub->num_rects + sub->num_rects/16 + 1;
     +-        } else
     +-            new_nb = UINT_MAX;
     +-    } else if (!nb_rect_allocated)
     +-        new_nb = sub->num_rects + 1;
     +-
     +-    if (new_nb) {
     +-        rects = av_realloc_array(rects, new_nb, sizeof(*sub->rects));
     +-        if (!rects)
     +-            return AVERROR(ENOMEM);
     +-        if (nb_rect_allocated)
     +-            *nb_rect_allocated = new_nb;
     +-        sub->rects = rects;
     +-    }
     +-
      -    rect       = av_mallocz(sizeof(*rect));
      -    if (!rect)
      -        return AVERROR(ENOMEM);
     @@ libavutil/ass.c: char *ff_ass_get_dialog(int readorder, int layer, const char *s
      -    return 0;
      -}
      -
     +-int ff_ass_add_rect(AVSubtitle *sub, const char *dialog,
     +-                    int readorder, int layer, const char *style,
     +-                    const char *speaker)
     ++char *avpriv_ass_get_dialog_ex(int readorder, int layer, const char *style,
     ++                        const char *speaker, int margin_l, int margin_r,
     ++                        int margin_v, const char *text)
     + {
     +-    return ff_ass_add_rect2(sub, dialog, readorder, layer, style, speaker, NULL);
     +-}
     +-
      -void ff_ass_decoder_flush(AVCodecContext *avctx)
      -{
      -    FFASSDecoderContext *s = avctx->priv_data;
     @@ libavutil/ass_internal.h (new)
      
       ## libavcodec/ass_split.c => libavutil/ass_split.c ##
      @@
     - #include "libavutil/common.h"
       #include "libavutil/error.h"
     + #include "libavutil/macros.h"
       #include "libavutil/mem.h"
      -#include "ass_split.h"
      +#include "ass_split_internal.h"
  6:  c6fb78e038 !  7:  09d8cf7880 avcodec/subtitles: Replace deprecated enum values
     @@ libavcodec/ass.h: static inline int avpriv_ass_add_rect(AVSubtitle *sub, const c
               return AVERROR(ENOMEM);
      
       ## libavcodec/assdec.c ##
     -@@ libavcodec/assdec.c: static int ass_decode_frame(AVCodecContext *avctx, void *data, int *got_sub_ptr,
     +@@ libavcodec/assdec.c: static int ass_decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
           if (!sub->rects[0])
               return AVERROR(ENOMEM);
           sub->num_rects = 1;
     @@ libavcodec/dvdsubenc.c: static int encode_dvd_subtitles(AVCodecContext *avctx,
               }
      
       ## libavcodec/pgssubdec.c ##
     -@@ libavcodec/pgssubdec.c: static int display_end_segment(AVCodecContext *avctx, void *data,
     +@@ libavcodec/pgssubdec.c: static int display_end_segment(AVCodecContext *avctx, AVSubtitle *sub,
               if (!rect)
                   return AVERROR(ENOMEM);
               sub->rects[sub->num_rects++] = rect;
     @@ libavcodec/pgssubdec.c: static int display_end_segment(AVCodecContext *avctx, vo
               object = find_object(ctx->presentation.objects[i].id, &ctx->objects);
      
       ## libavcodec/xsubdec.c ##
     -@@ libavcodec/xsubdec.c: static int decode_frame(AVCodecContext *avctx, void *data, int *got_sub_ptr,
     +@@ libavcodec/xsubdec.c: static int decode_frame(AVCodecContext *avctx, AVSubtitle *sub,
           sub->num_rects = 1;
           rect->x = x; rect->y = y;
           rect->w = w; rect->h = h;
  7:  3d8673919f =  8:  897299bf7f fftools/play,probe: Adjust for subtitle changes
  8:  ba8b675326 !  9:  ca580c6d21 avfilter/subtitles: Add subtitles.c for subtitle frame allocation
     @@ libavfilter/Makefile: OBJS = allfilters.o
              graphdump.o                                                      \
              graphparser.o                                                    \
      +       subtitles.o                                                      \
     +        version.o                                                        \
              video.o                                                          \
       
     - OBJS-$(HAVE_THREADS)                         += pthread.o
      
       ## libavfilter/avfilter.c ##
      @@
     @@ libavfilter/avfilter.c
       #include "internal.h"
      +#include "subtitles.h"
       
     - #include "libavutil/ffversion.h"
     - const char av_filter_ffversion[] = "FFmpeg version " FFMPEG_VERSION;
     + static void tlog_ref(void *ctx, AVFrame *ref, int end)
     + {
      @@ libavfilter/avfilter.c: int ff_inlink_make_frame_writable(AVFilterLink *link, AVFrame **rframe)
           case AVMEDIA_TYPE_AUDIO:
               out = ff_get_audio_buffer(link, frame->nb_samples);
  9:  eb09db0e00 ! 10:  0781e974a2 avfilter/avfilter: Handle subtitle frames
     @@ libavfilter/avfilter.c: static void tlog_ref(void *ctx, AVFrame *ref, int end)
           }
       
           ff_tlog(ctx, "]%s", end ? "\n" : "");
     +@@ libavfilter/avfilter.c: int ff_filter_frame(AVFilterLink *link, AVFrame *frame)
     +             av_assert1(frame->width               == link->w);
     +             av_assert1(frame->height               == link->h);
     +         }
     ++    } else if (link->type == AVMEDIA_TYPE_SUBTITLE) {
     ++        if (frame->format != link->format) {
     ++            av_log(link->dst, AV_LOG_WARNING, "Subtitle format change from %d to %d\n", link->format, frame->format);
     ++        }
     +     } else {
     +         if (frame->format != link->format) {
     +             av_log(link->dst, AV_LOG_ERROR, "Format change is not supported\n");
      
       ## libavfilter/avfilter.h ##
      @@
     @@ libavfilter/avfilter.h
      +#include "libavutil/subfmt.h"
       #include "libavutil/rational.h"
       
     - #include "libavfilter/version.h"
     + #include "libavfilter/version_major.h"
      @@ libavfilter/avfilter.h: typedef struct AVFilter {
                * and outputs use the same sample rate and channel count/layout.
                */
     @@ libavfilter/formats.c
       #include "libavutil/avassert.h"
       #include "libavutil/channel_layout.h"
       #include "libavutil/common.h"
     -@@ libavfilter/formats.c: int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout)
     -     return 0;
     - }
     - 
     -+int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt)
     -+{
     -+    ADD_FORMAT(avff, fmt, ff_formats_unref, int, formats, nb_formats);
     -+    return 0;
     -+}
     -+
     - AVFilterFormats *ff_make_formats_list_singleton(int fmt)
     - {
     -     int fmts[2] = { fmt, -1 };
      @@ libavfilter/formats.c: AVFilterFormats *ff_all_formats(enum AVMediaType type)
                       return NULL;
                   fmt++;
               }
      +    } else if (type == AVMEDIA_TYPE_SUBTITLE) {
     -+        if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_BITMAP) < 0)
     ++        if (ff_add_format(&ret, AV_SUBTITLE_FMT_BITMAP) < 0)
      +            return NULL;
     -+        if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_ASS) < 0)
     ++        if (ff_add_format(&ret, AV_SUBTITLE_FMT_ASS) < 0)
      +            return NULL;
     -+        if (ff_add_subtitle_type(&ret, AV_SUBTITLE_FMT_TEXT) < 0)
     ++        if (ff_add_format(&ret, AV_SUBTITLE_FMT_TEXT) < 0)
      +            return NULL;
           }
       
     @@ libavfilter/formats.c: int ff_default_query_formats(AVFilterContext *ctx)
           /* Intended fallthrough */
      
       ## libavfilter/formats.h ##
     -@@ libavfilter/formats.h: int ff_set_common_formats_from_list(AVFilterContext *ctx, const int *fmts);
     - av_warn_unused_result
     - int ff_add_channel_layout(AVFilterChannelLayouts **l, uint64_t channel_layout);
     +@@ libavfilter/formats.h: av_warn_unused_result
     + int ff_add_channel_layout(AVFilterChannelLayouts **l,
     +                           const AVChannelLayout *channel_layout);
       
      +av_warn_unused_result
      +int ff_add_subtitle_type(AVFilterFormats **avff, int64_t fmt);
 10:  b31a991320 = 11:  d9d9f42558 avfilter/avfilter: Fix hardcoded input index
 11:  59abea7693 ! 12:  af69a4b321 avfilter/sbuffer: Add sbuffersrc and sbuffersink filters
     @@ libavfilter/buffersink.c: static int asink_query_formats(AVFilterContext *ctx)
      +    CHECK_LIST_SIZE(subtitle_types)
      +    if (buf->subtitle_types_size) {
      +        for (i = 0; i < NB_ITEMS(buf->subtitle_types); i++)
     -+            if ((ret = ff_add_subtitle_type(&formats, buf->subtitle_types[i])) < 0)
     ++            if ((ret = ff_add_format(&formats, buf->subtitle_types[i])) < 0)
      +                return ret;
      +        if ((ret = ff_set_common_formats(ctx, formats)) < 0)
      +            return ret;
     @@ libavfilter/buffersrc.c
       typedef struct BufferSourceContext {
           const AVClass    *class;
      @@ libavfilter/buffersrc.c: typedef struct BufferSourceContext {
     -     uint64_t channel_layout;
           char    *channel_layout_str;
     +     AVChannelLayout ch_layout;
       
      +    /* subtitle only */
      +    enum AVSubtitleType subtitle_type;
     @@ libavfilter/buffersrc.c: typedef struct BufferSourceContext {
           int eof;
       } BufferSourceContext;
       
     -@@ libavfilter/buffersrc.c: int av_buffersrc_parameters_set(AVFilterContext *ctx, AVBufferSrcParameters *par
     -         if (param->channel_layout)
     -             s->channel_layout = param->channel_layout;
     +@@ libavfilter/buffersrc.c: FF_ENABLE_DEPRECATION_WARNINGS
     +                 return ret;
     +         }
               break;
      +    case AVMEDIA_TYPE_SUBTITLE:
      +        s->subtitle_type = param->format;
     @@ libavfilter/buffersrc.c: int av_buffersrc_parameters_set(AVFilterContext *ctx, A
           default:
               return AVERROR_BUG;
           }
     -@@ libavfilter/buffersrc.c: int attribute_align_arg av_buffersrc_add_frame_flags(AVFilterContext *ctx, AVFra
     -             CHECK_AUDIO_PARAM_CHANGE(ctx, s, frame->sample_rate, frame->channel_layout,
     -                                      frame->channels, frame->format, frame->pts);
     +@@ libavfilter/buffersrc.c: FF_ENABLE_DEPRECATION_WARNINGS
     +             CHECK_AUDIO_PARAM_CHANGE(ctx, s, frame->sample_rate, frame->ch_layout,
     +                                      frame->format, frame->pts);
                   break;
      +        case AVMEDIA_TYPE_SUBTITLE:
      +            break;
     @@ libavfilter/buffersrc.c: static const AVOption abuffer_options[] = {
       static av_cold int init_audio(AVFilterContext *ctx)
       {
           BufferSourceContext *s = ctx->priv;
     -@@ libavfilter/buffersrc.c: static av_cold int init_audio(AVFilterContext *ctx)
     +@@ libavfilter/buffersrc.c: FF_ENABLE_DEPRECATION_WARNINGS
           return ret;
       }
       
     @@ libavfilter/buffersrc.c: static int query_formats(AVFilterContext *ctx)
               return AVERROR(EINVAL);
           }
      @@ libavfilter/buffersrc.c: static int config_props(AVFilterLink *link)
     -         if (!c->channel_layout)
     -             c->channel_layout = link->channel_layout;
     +                 return ret;
     +         }
               break;
      +    case AVMEDIA_TYPE_SUBTITLE:
      +        link->format = c->subtitle_type;
     @@ libavfilter/buffersrc.c: const AVFilter ff_asrc_abuffer = {
           .priv_class = &abuffer_class,
       };
      +
     -+static const AVFilterPad ssrc_sbuffer_outputs[] = {
     ++static const AVFilterPad avfilter_ssrc_sbuffer_outputs[] = {
      +    {
      +        .name          = "default",
      +        .type          = AVMEDIA_TYPE_SUBTITLE,
     @@ libavfilter/buffersrc.c: const AVFilter ff_asrc_abuffer = {
      +    .uninit    = uninit,
      +
      +    .inputs    = NULL,
     -+    FILTER_OUTPUTS(ssrc_sbuffer_outputs),
     ++    FILTER_OUTPUTS(avfilter_ssrc_sbuffer_outputs),
      +    FILTER_QUERY_FUNC(query_formats),
      +    .priv_class = &sbuffer_class,
      +};
 12:  867b60c60d ! 13:  f7e5b590a2 avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters
     @@ doc/filters.texi: tools.
      +This filter replaces the previous "sub2video" hack which did the conversion implicitly and up-front as subtitle filtering wasn't possible at that time.
      +To retain compatibility with earlier sub2video command lines, this filter is being auto-inserted in those cases.
      +
     -+For overlaying graphicsal subtitles it is recommended to use the 'overlay_graphicsubs' filter which is more efficient and takes less processing resources.
     ++For overlaying graphicsal subtitles it is recommended to use the 'overlaygraphicsubs' filter which is more efficient and takes less processing resources.
      +
      +This filter is still useful in cases where the overlay is done with hardware acceleration (e.g. overlay_qsv, overlay_vaapi, overlay_cuda) for preparing the overlay frames.
      +
     @@ doc/filters.texi: tools.
      +@itemize
      +@item
      +Overlay PGS subtitles
     -+(not recommended - better use overlay_graphicsubs)
     ++(not recommended - better use overlaygraphicsubs)
      +@example
      +ffmpeg -i "https://streams.videolan.org/samples/sub/PGS/Girl_With_The_Dragon_Tattoo_2%3A23%3A56.mkv" -filter_complex "[0:1]graphicsub2video[subs];[0:0][subs]overlay" output.mp4
      +@end example
     @@ libavfilter/Makefile: OBJS-$(CONFIG_OVERLAY_OPENCL_FILTER)         += vf_overlay
       OBJS-$(CONFIG_PAD_OPENCL_FILTER)             += vf_pad_opencl.o opencl.o opencl/pad.o
      
       ## libavfilter/allfilters.c ##
     -@@ libavfilter/allfilters.c: extern const AVFilter ff_vf_oscilloscope;
     - extern const AVFilter ff_vf_overlay;
     - extern const AVFilter ff_vf_overlay_opencl;
     - extern const AVFilter ff_vf_overlay_qsv;
     -+extern const AVFilter ff_vf_overlaygraphicsubs;
     +@@ libavfilter/allfilters.c: extern const AVFilter ff_vf_overlay_qsv;
       extern const AVFilter ff_vf_overlay_vaapi;
       extern const AVFilter ff_vf_overlay_vulkan;
       extern const AVFilter ff_vf_overlay_cuda;
     ++extern const AVFilter ff_vf_overlaygraphicsubs;
     + extern const AVFilter ff_vf_owdenoise;
     + extern const AVFilter ff_vf_pad;
     + extern const AVFilter ff_vf_pad_opencl;
      @@ libavfilter/allfilters.c: extern const AVFilter ff_avf_showvolume;
       extern const AVFilter ff_avf_showwaves;
       extern const AVFilter ff_avf_showwavespic;
     @@ libavfilter/allfilters.c: extern const AVFilter ff_avf_showvolume;
      +extern const AVFilter ff_svf_graphicsub2video;
       
       /* multimedia sources */
     - extern const AVFilter ff_avsrc_amovie;
     + extern const AVFilter ff_avsrc_avsynctest;
      
       ## libavfilter/vf_overlaygraphicsubs.c (new) ##
      @@
 13:  f88a5667e1 ! 14:  4c8092357f avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters
     @@ configure: overlay_qsv_filter_deps="libmfx"
       owdenoise_filter_deps="gpl"
       pad_opencl_filter_deps="opencl"
       pan_filter_deps="swresample"
     -@@ configure: superequalizer_filter_deps="avcodec"
     - superequalizer_filter_select="rdft"
     - surround_filter_deps="avcodec"
     - surround_filter_select="rdft"
     +@@ configure: stereo3d_filter_deps="gpl"
     + subtitles_filter_deps="avformat avcodec libass"
     + super2xsai_filter_deps="gpl"
     + pixfmts_super2xsai_test_deps="super2xsai_filter"
      +textsub2video_filter_deps="avcodec libass"
       tinterlace_filter_deps="gpl"
       tinterlace_merge_test_deps="tinterlace_filter"
     @@ doc/filters.texi: Overlay PGS subtitles
      +
      +Converts text subtitles to video frames.
      +
     -+For overlaying text subtitles onto video frames it is recommended to use the overlay_textsubs filter.
     ++For overlaying text subtitles onto video frames it is recommended to use the overlaytextsubs filter.
      +The textsub2video is useful for for creating transparent text-frames when overlay is done via hw acceleration
      +
      +Inputs:
     @@ libavfilter/Makefile: OBJS-$(CONFIG_SWAPRECT_FILTER)               += vf_swaprec
       OBJS-$(CONFIG_THUMBNAIL_FILTER)              += vf_thumbnail.o
      
       ## libavfilter/allfilters.c ##
     -@@ libavfilter/allfilters.c: extern const AVFilter ff_vf_oscilloscope;
     - extern const AVFilter ff_vf_overlay;
     - extern const AVFilter ff_vf_overlay_opencl;
     - extern const AVFilter ff_vf_overlay_qsv;
     --extern const AVFilter ff_vf_overlaygraphicsubs;
     --extern const AVFilter ff_vf_overlay_vaapi;
     +@@ libavfilter/allfilters.c: extern const AVFilter ff_vf_overlay_vaapi;
       extern const AVFilter ff_vf_overlay_vulkan;
       extern const AVFilter ff_vf_overlay_cuda;
     -+extern const AVFilter ff_vf_overlaygraphicsubs;
     + extern const AVFilter ff_vf_overlaygraphicsubs;
      +extern const AVFilter ff_vf_overlaytextsubs;
     -+extern const AVFilter ff_vf_overlay_vaapi;
       extern const AVFilter ff_vf_owdenoise;
       extern const AVFilter ff_vf_pad;
       extern const AVFilter ff_vf_pad_opencl;
     @@ libavfilter/allfilters.c: extern const AVFilter ff_avf_showwaves;
      +extern const AVFilter ff_svf_textsub2video;
       
       /* multimedia sources */
     - extern const AVFilter ff_avsrc_amovie;
     + extern const AVFilter ff_avsrc_avsynctest;
      
       ## libavfilter/vf_overlaytextsubs.c (new) ##
      @@
     @@ libavfilter/vf_overlaytextsubs.c (new)
      + * overlay text subtitles on top of a video frame
      + */
      +
     ++#include "config_components.h"
     ++
      +#include <ass/ass.h>
      +#include "libavutil/ass_internal.h"
      +#include "libavutil/thread.h"
 14:  25cda21970 ! 15:  8fdbdf7c5f avfilter/textmod: Add textmod, censor and show_speaker filters
     @@ doc/filters.texi: ffmpeg -i "http://streams.videolan.org/samples/sub/SSA/subtitl
      +@item
      +Prepend speaker names with blue text, smooth edges and blend/overlay that onto the video.
      +@example
     -+ffmpeg -i INPUT -filter_complex "showspeaker=format=colon:style='@{\\c&HDD0000&\\be1@}',[0:v]overlay_textsubs"
     ++ffmpeg -i INPUT -filter_complex "showspeaker=format=colon:style='@{\\c&HDD0000&\\be1@}',[0:v]overlaytextsubs"
      +@end example
      +@end itemize
      +
 15:  296f1f697b = 16:  d44b22f15b avfilter/stripstyles: Add stripstyles filter
 16:  509d6c67ba = 17:  28d75dc982 avfilter/splitcc: Add splitcc filter for closed caption handling
 17:  1d7acba39f ! 18:  42d1d1c819 avfilter/graphicsub2text: Add new graphicsub2text filter (OCR)
     @@ doc/filters.texi: ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.m
       Renders graphic subtitles as video frames.
      
       ## libavfilter/Makefile ##
     -@@ libavfilter/Makefile: OBJS-$(CONFIG_GBLUR_VULKAN_FILTER)           += vf_gblur_vulkan.o vulkan.o vulka
     +@@ libavfilter/Makefile: OBJS-$(CONFIG_GBLUR_FILTER)                  += vf_gblur.o
     + OBJS-$(CONFIG_GBLUR_VULKAN_FILTER)           += vf_gblur_vulkan.o vulkan.o vulkan_filter.o
       OBJS-$(CONFIG_GEQ_FILTER)                    += vf_geq.o
       OBJS-$(CONFIG_GRADFUN_FILTER)                += vf_gradfun.o
     - OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER)       += vf_overlaygraphicsubs.o framesync.o
      +OBJS-$(CONFIG_GRAPHICSUB2TEXT_FILTER)        += sf_graphicsub2text.o
     -+OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER)       += vf_overlaygraphicsubs.o framesync.o
     + OBJS-$(CONFIG_GRAPHICSUB2VIDEO_FILTER)       += vf_overlaygraphicsubs.o framesync.o
       OBJS-$(CONFIG_GRAPHMONITOR_FILTER)           += f_graphmonitor.o
       OBJS-$(CONFIG_GRAYWORLD_FILTER)              += vf_grayworld.o
     - OBJS-$(CONFIG_GREYEDGE_FILTER)               += vf_colorconstancy.o
      
       ## libavfilter/allfilters.c ##
      @@ libavfilter/allfilters.c: extern const AVFilter ff_avf_showwaves;
 18:  244dd6de33 ! 19:  7095e8aa26 avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles
     @@ doc/filters.texi: Set the rendering margin in pixels.
      +inside the video area without scaling, then overlay
      +subtitles onto the video.
      +@example
     -+ffmpeg -y -loglevel verbose -ss 32 -i "https://streams.videolan.org/samples/sub/largeres_vobsub.mkv" -filter_complex "[0:0]scale=1920x400,setsar=1[vid1];[0:10]subscale=w=1920:h=400:scale_mode=none:margin_h=50:arrange_h=margin_no_scale:arrange_v=margin_no_scale:margin_v=50:num_colors=256[sub1];[vid1][sub1]overlay_graphicsubs" -c:v libx265 output.ts
     ++ffmpeg -y -loglevel verbose -ss 32 -i "https://streams.videolan.org/samples/sub/largeres_vobsub.mkv" -filter_complex "[0:0]scale=1920x400,setsar=1[vid1];[0:10]subscale=w=1920:h=400:scale_mode=none:margin_h=50:arrange_h=margin_no_scale:arrange_v=margin_no_scale:margin_v=50:num_colors=256[sub1];[vid1][sub1]overlaygraphicsubs" -c:v libx265 output.ts
      +@end example
      +@item
      +Scale both, a video and its embedded VOB subs, then encode them as separate streams into an MKV.
 19:  a87812bd3c ! 20:  697939451e avfilter/subfeed: add subtitle feed filter
     @@ libavfilter/sf_subfeed.c (new)
      +        next_pts = 0;
      +
      +    if (s->next_pts_offset) {
     ++        av_log(outlink->src, AV_LOG_VERBOSE, "Subtracting next_pts_offset: %"PRId64" \n", s->next_pts_offset);
      +        next_pts -= s->next_pts_offset;
      +        s->next_pts_offset = 0;
      +    }
     @@ libavfilter/sf_subfeed.c (new)
      +        const AVFrame *current_frame = ff_framequeue_peek(&s->fifo, 0);
      +        const int64_t sub_end_time   = current_frame->subtitle_timing.start_pts + current_frame->subtitle_timing.duration;
      +
     -+        if (next_pts > sub_end_time) {
     -+            AVFrame *remove_frame = ff_framequeue_take(&s->fifo);
     -+            av_frame_free(&remove_frame);
     -+
     -+            if (ff_framequeue_queued_frames(&s->fifo)) {
     ++        if (ff_framequeue_queued_frames(&s->fifo) > 1) {
     ++            const AVFrame *next_frame = ff_framequeue_peek(&s->fifo, 1);
     ++            if (next_pts + interval > next_frame->subtitle_timing.start_pts) {
     ++                AVFrame *remove_frame = ff_framequeue_take(&s->fifo);
     ++                av_frame_free(&remove_frame);
      +                s->current_frame_isnew = 1;
      +                goto retry;
      +            }
      +        }
     ++
     ++        if (next_pts > sub_end_time) {
     ++            AVFrame *remove_frame = ff_framequeue_take(&s->fifo);
     ++            av_frame_free(&remove_frame);
     ++            s->current_frame_isnew = 1;
     ++            goto retry;
     ++        }
      +    }
      +
      +    if (ff_framequeue_queued_frames(&s->fifo)) {
     @@ libavfilter/sf_subfeed.c (new)
      +            s->current_frame_isnew = 0;
      +            s->recent_subtitle_pts = out->subtitle_timing.start_pts;
      +
     ++            av_log(outlink->src, AV_LOG_DEBUG, "Output1 frame pts: %"PRId64"  subtitle_pts: %"PRId64"  repeat_frame: %d\n",
     ++                out->pts, out->subtitle_timing.start_pts, out->repeat_sub);
     ++
      +            return ff_filter_frame(outlink, out);
      +        }
      +    }
     @@ libavfilter/sf_subfeed.c (new)
      +    out->repeat_sub = 1;
      +    out->subtitle_timing.start_pts = s->recent_subtitle_pts;
      +
     ++    av_log(outlink->src, AV_LOG_DEBUG, "Output2 frame pts: %"PRId64"  subtitle_pts: %"PRId64"  repeat_frame: %d\n",
     ++        out->pts, out->subtitle_timing.start_pts, out->repeat_sub);
     ++
      +    return ff_filter_frame(outlink, out);
      +}
      +
     @@ libavfilter/sf_subfeed.c (new)
      +    SubFeedContext *s           = inlink->dst->priv;
      +    AVFilterLink *outlink       = inlink->dst->outputs[0];
      +    const int64_t index         = (int64_t)ff_framequeue_queued_frames(&s->fifo) - 1;
     ++    size_t nb_queued_frames;
      +    int ret = 0;
      +
     ++    av_log(ctx, AV_LOG_VERBOSE, "frame.pts: %"PRId64" (AVTB: %"PRId64") -  subtitle_timing.start_pts: %"PRId64" subtitle_timing.duration: %"PRId64" - format: %d\n",
     ++        frame->pts, av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q), frame->subtitle_timing.start_pts, frame->subtitle_timing.duration, frame->format);
     ++
      +    frame->pts = av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q);
      +
      +    if (index < 0) {
     @@ libavfilter/sf_subfeed.c (new)
      +    } else if (s->fix_durations || s->fix_overlap) {
      +        AVFrame *previous_frame = ff_framequeue_peek(&s->fifo, index);
      +        const int64_t pts_diff = frame->subtitle_timing.start_pts - previous_frame->subtitle_timing.start_pts;
     ++        nb_queued_frames = ff_framequeue_queued_frames(&s->fifo);
      +
     -+        if (s->fix_durations && pts_diff > 0 && previous_frame->subtitle_timing.duration > ms_to_avtb(29000))
     -+                    previous_frame->subtitle_timing.duration = frame->subtitle_timing.start_pts - previous_frame->subtitle_timing.start_pts;
     ++        if (s->fix_durations && pts_diff > 0 && previous_frame->subtitle_timing.duration > ms_to_avtb(29000)) {
     ++            av_log(ctx, AV_LOG_VERBOSE, "Previous frame (index #%"PRId64") has a duration of %"PRId64" ms, setting to  %"PRId64" ms\n",
     ++                index, avtb_to_ms(previous_frame->subtitle_timing.duration), avtb_to_ms(frame->subtitle_timing.start_pts - previous_frame->subtitle_timing.start_pts));
     ++            previous_frame->subtitle_timing.duration = frame->subtitle_timing.start_pts - previous_frame->subtitle_timing.start_pts;
     ++        }
     ++
     ++        if (s->fix_overlap && pts_diff > 0 && previous_frame->subtitle_timing.duration > pts_diff) {
     ++            av_log(ctx, AV_LOG_VERBOSE, "Detected overlap from previous frame (index #%"PRId64") which had a duration of %"PRId64" ms, setting to the pts_diff which is %"PRId64" ms\n",
     ++                index, avtb_to_ms(previous_frame->subtitle_timing.duration), avtb_to_ms(pts_diff));
     ++            previous_frame->subtitle_timing.duration = pts_diff;
     ++        }
      +
     -+        if (s->fix_overlap && pts_diff > 0 && previous_frame->subtitle_timing.duration > pts_diff)
     -+                    previous_frame->subtitle_timing.duration = pts_diff;
     ++        if (pts_diff <= 0) {
     ++            av_log(ctx, AV_LOG_WARNING, "The pts_diff to the previous frame (index #%"PRId64")  is <= 0: %"PRId64" ms. The previous frame duration is %"PRId64" ms.\n",
     ++                index, avtb_to_ms(pts_diff),  avtb_to_ms(previous_frame->subtitle_timing.duration));
     ++        }
      +    }
      +
      +    ff_framequeue_add(&s->fifo, frame);
      +
     -+    if (index > 3)
     -+        av_log(ctx, AV_LOG_WARNING, "frame queue count: %d\n", (int)index);
     ++    nb_queued_frames = ff_framequeue_queued_frames(&s->fifo);
     ++
     ++    if (nb_queued_frames > 3)
     ++        av_log(ctx, AV_LOG_WARNING, "frame queue count: %zu\n", nb_queued_frames);
      +
     -+    if (s->mode == FM_FORWARD && ff_framequeue_queued_frames(&s->fifo)) {
     ++    if (s->mode == FM_FORWARD && nb_queued_frames) {
      +
      +        AVFrame *first_frame = ff_framequeue_peek(&s->fifo, 0);
      +
     -+        if (s->fix_overlap && ff_framequeue_queued_frames(&s->fifo) < 2)
     -+            return 0;
     ++        if (s->fix_overlap && nb_queued_frames < 2) {
     ++          av_log(ctx, AV_LOG_VERBOSE, "Return no frame since we have less than 2\n");
     ++          return 0;
     ++        }
      +
     -+        if (s->fix_durations && first_frame->subtitle_timing.duration > ms_to_avtb(29000))
     ++        if (s->fix_durations && first_frame->subtitle_timing.duration > ms_to_avtb(29000)) {
     ++            av_log(ctx, AV_LOG_VERBOSE, "Return no frame because first frame duration is %"PRId64" ms\n", avtb_to_ms(first_frame->subtitle_timing.duration));
      +            return 0;
     ++        }
      +
      +        first_frame = ff_framequeue_take(&s->fifo);
      +        return ff_filter_frame(outlink, first_frame);
 20:  aea8acb057 ! 21:  32e9af0806 avcodec/subtitles: Migrate subtitle encoders to frame-based API
     @@ Commit message
      
       ## libavcodec/assenc.c ##
      @@
     - #include <string.h>
       
       #include "avcodec.h"
     + #include "codec_internal.h"
      +#include "encode.h"
       #include "libavutil/ass_internal.h"
     - #include "internal.h"
       #include "libavutil/avstring.h"
       #include "libavutil/internal.h"
       #include "libavutil/mem.h"
     @@ libavcodec/assenc.c
       }
       
       #if CONFIG_SSA_ENCODER
     - const AVCodec ff_ssa_encoder = {
     --    .name         = "ssa",
     --    .long_name    = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     --    .type         = AVMEDIA_TYPE_SUBTITLE,
     --    .id           = AV_CODEC_ID_ASS,
     + const FFCodec ff_ssa_encoder = {
     +-    .p.name       = "ssa",
     +-    .p.long_name  = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     +-    .p.type       = AVMEDIA_TYPE_SUBTITLE,
     +-    .p.id         = AV_CODEC_ID_ASS,
      -    .init         = ass_encode_init,
     --    .encode_sub   = ass_encode_frame,
     -+    .name           = "ssa",
     -+    .long_name      = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     -+    .type           = AVMEDIA_TYPE_SUBTITLE,
     -+    .id             = AV_CODEC_ID_ASS,
     +-    FF_CODEC_ENCODE_SUB_CB(ass_encode_frame),
     ++    .p.name           = "ssa",
     ++    .p.long_name      = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     ++    .p.type           = AVMEDIA_TYPE_SUBTITLE,
     ++    .p.id             = AV_CODEC_ID_ASS,
      +    .priv_data_size = sizeof(AssEncContext),
      +    .init           = ass_encode_init,
      +    .close          = ass_encode_close,
     -+    .receive_packet = ass_receive_packet,
     ++    FF_CODEC_RECEIVE_PACKET_CB(ass_receive_packet),
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
       #endif
       
       #if CONFIG_ASS_ENCODER
     - const AVCodec ff_ass_encoder = {
     --    .name         = "ass",
     --    .long_name    = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     --    .type         = AVMEDIA_TYPE_SUBTITLE,
     --    .id           = AV_CODEC_ID_ASS,
     + const FFCodec ff_ass_encoder = {
     +-    .p.name       = "ass",
     +-    .p.long_name  = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     +-    .p.type       = AVMEDIA_TYPE_SUBTITLE,
     +-    .p.id         = AV_CODEC_ID_ASS,
      -    .init         = ass_encode_init,
     --    .encode_sub   = ass_encode_frame,
     -+    .name           = "ass",
     -+    .long_name      = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     -+    .type           = AVMEDIA_TYPE_SUBTITLE,
     +-    FF_CODEC_ENCODE_SUB_CB(ass_encode_frame),
     ++    .p.name           = "ass",
     ++    .p.long_name      = NULL_IF_CONFIG_SMALL("ASS (Advanced SubStation Alpha) subtitle"),
     ++    .p.type           = AVMEDIA_TYPE_SUBTITLE,
      +    .priv_data_size = sizeof(AssEncContext),
     -+    .id             = AV_CODEC_ID_ASS,
     ++    .p.id             = AV_CODEC_ID_ASS,
      +    .init           = ass_encode_init,
      +    .close          = ass_encode_close,
     -+    .receive_packet = ass_receive_packet,
     ++    FF_CODEC_RECEIVE_PACKET_CB(ass_receive_packet),
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
       #endif
     @@ libavcodec/avcodec.h: void av_parser_close(AVCodecParserContext *s);
        * @}
        */
      
     + ## libavcodec/codec_internal.h ##
     +@@ libavcodec/codec_internal.h: enum FFCodecType {
     +     /* The codec is an encoder using the encode callback;
     +      * audio and video codecs only. */
     +     FF_CODEC_CB_TYPE_ENCODE,
     +-    /* The codec is an encoder using the encode_sub callback;
     +-     * subtitle codecs only. */
     +-    FF_CODEC_CB_TYPE_ENCODE_SUB,
     +     /* The codec is an encoder using the receive_packet callback;
     +      * audio and video codecs only. */
     +     FF_CODEC_CB_TYPE_RECEIVE_PACKET,
     +@@ libavcodec/codec_internal.h: typedef struct FFCodec {
     +          */
     +         int (*encode)(struct AVCodecContext *avctx, struct AVPacket *avpkt,
     +                       const struct AVFrame *frame, int *got_packet_ptr);
     +-        /**
     +-         * Encode subtitles to a raw buffer.
     +-         * cb is in this state if cb_type is FF_CODEC_CB_TYPE_ENCODE_SUB.
     +-         */
     +-        int (*encode_sub)(struct AVCodecContext *avctx, uint8_t *buf,
     +-                          int buf_size, const struct AVSubtitle *sub);
     +         /**
     +          * Encode API with decoupled frame/packet dataflow.
     +          * cb is in this state if cb_type is FF_CODEC_CB_TYPE_RECEIVE_PACKET.
     +@@ libavcodec/codec_internal.h: typedef struct FFCodec {
     + #define FF_CODEC_ENCODE_CB(func)                          \
     +     .cb_type           = FF_CODEC_CB_TYPE_ENCODE,         \
     +     .cb.encode         = (func)
     +-#define FF_CODEC_ENCODE_SUB_CB(func)                      \
     +-    .cb_type           = FF_CODEC_CB_TYPE_ENCODE_SUB,     \
     +-    .cb.encode_sub     = (func)
     + #define FF_CODEC_RECEIVE_PACKET_CB(func)                  \
     +     .cb_type           = FF_CODEC_CB_TYPE_RECEIVE_PACKET, \
     +     .cb.receive_packet = (func)
     +
       ## libavcodec/dvbsubenc.c ##
      @@
     -  */
       #include "avcodec.h"
       #include "bytestream.h"
     + #include "codec_internal.h"
      +#include "encode.h"
       #include "libavutil/colorspace.h"
       
     @@ libavcodec/dvbsubenc.c: static int dvbsub_encode(AVCodecContext *avctx, uint8_t
      +    return 0;
       }
       
     - const AVCodec ff_dvbsub_encoder = {
     -@@ libavcodec/dvbsubenc.c: const AVCodec ff_dvbsub_encoder = {
     -     .type           = AVMEDIA_TYPE_SUBTITLE,
     -     .id             = AV_CODEC_ID_DVB_SUBTITLE,
     + const FFCodec ff_dvbsub_encoder = {
     +@@ libavcodec/dvbsubenc.c: const FFCodec ff_dvbsub_encoder = {
     +     .p.type         = AVMEDIA_TYPE_SUBTITLE,
     +     .p.id           = AV_CODEC_ID_DVB_SUBTITLE,
           .priv_data_size = sizeof(DVBSubtitleContext),
     --    .encode_sub     = dvbsub_encode,
     -+    .encode2        = dvbsub_encode,
     +-    FF_CODEC_ENCODE_SUB_CB(dvbsub_encode),
     ++    FF_CODEC_ENCODE_CB(dvbsub_encode),
       };
      
       ## libavcodec/dvdsubenc.c ##
      @@
     -  */
       #include "avcodec.h"
       #include "bytestream.h"
     + #include "codec_internal.h"
      +#include "encode.h"
       #include "internal.h"
       #include "libavutil/avassert.h"
     @@ libavcodec/dvdsubenc.c: static int dvdsub_init(AVCodecContext *avctx)
           return ret;
       }
       
     -@@ libavcodec/dvdsubenc.c: const AVCodec ff_dvdsub_encoder = {
     -     .type           = AVMEDIA_TYPE_SUBTITLE,
     -     .id             = AV_CODEC_ID_DVD_SUBTITLE,
     +@@ libavcodec/dvdsubenc.c: const FFCodec ff_dvdsub_encoder = {
     +     .p.type         = AVMEDIA_TYPE_SUBTITLE,
     +     .p.id           = AV_CODEC_ID_DVD_SUBTITLE,
           .init           = dvdsub_init,
     --    .encode_sub     = dvdsub_encode,
     -+    .encode2        = dvdsub_encode,
     -     .priv_class     = &dvdsubenc_class,
     +-    FF_CODEC_ENCODE_SUB_CB(dvdsub_encode),
     ++    FF_CODEC_ENCODE_CB(dvdsub_encode),
     +     .p.priv_class   = &dvdsubenc_class,
           .priv_data_size = sizeof(DVDSubtitleContext),
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
      
     @@ libavcodec/encode.c: fail:
      +int avcodec_encode_subtitle(AVCodecContext *avctx, uint8_t *buf, int buf_size, const AVSubtitle *sub)
       {
      -    int ret;
     -+    int ret = 0, got_packet = 0;
     ++    int ret = 0;
      +    AVFrame *frame = NULL;
      +    AVPacket* avpkt = NULL;
      +
     @@ libavcodec/encode.c: fail:
               return -1;
           }
       
     --    ret = avctx->codec->encode_sub(avctx, buf, buf_size, sub);
     +-    ret = ffcodec(avctx->codec)->cb.encode_sub(avctx, buf, buf_size, sub);
      +    memset(buf, 0, buf_size);
      +    // Create a temporary frame for calling the regular api:
      +    frame = av_frame_alloc();
     @@ libavcodec/encode.c: fail:
      
       ## libavcodec/movtextenc.c ##
      @@
     - #include "libavutil/ass_split_internal.h"
       #include "libavutil/ass_internal.h"
       #include "bytestream.h"
     + #include "codec_internal.h"
      +#include "encode.h"
     - #include "internal.h"
       
       #define STYLE_FLAG_BOLD         (1<<0)
     + #define STYLE_FLAG_ITALIC       (1<<1)
      @@ libavcodec/movtextenc.c: typedef struct {
           AVCodecContext *avctx;
       
     @@ libavcodec/movtextenc.c: static const ASSCodesCallbacks mov_text_callbacks = {
       }
       
       #define OFFSET(x) offsetof(MovTextContext, x)
     -@@ libavcodec/movtextenc.c: const AVCodec ff_movtext_encoder = {
     +@@ libavcodec/movtextenc.c: const FFCodec ff_movtext_encoder = {
           .priv_data_size = sizeof(MovTextContext),
     -     .priv_class     = &mov_text_encoder_class,
     +     .p.priv_class   = &mov_text_encoder_class,
           .init           = mov_text_encode_init,
     --    .encode_sub     = mov_text_encode_frame,
     -+    .encode2        = mov_text_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(mov_text_encode_frame),
     ++    FF_CODEC_ENCODE_CB(mov_text_encode_frame),
           .close          = mov_text_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP,
       };
     @@ libavcodec/srtenc.c
      +#include "encode.h"
       #include "libavutil/avstring.h"
       #include "libavutil/bprint.h"
     - #include "libavutil/ass_split_internal.h"
     + #include "codec_internal.h"
      @@
       typedef struct {
           AVCodecContext *avctx;
     @@ libavcodec/srtenc.c: static const ASSCodesCallbacks text_callbacks = {
       }
       
       static int srt_encode_close(AVCodecContext *avctx)
     -@@ libavcodec/srtenc.c: const AVCodec ff_srt_encoder = {
     -     .id             = AV_CODEC_ID_SUBRIP,
     +@@ libavcodec/srtenc.c: const FFCodec ff_srt_encoder = {
     +     .p.id           = AV_CODEC_ID_SUBRIP,
           .priv_data_size = sizeof(SRTContext),
           .init           = srt_encode_init,
     --    .encode_sub     = srt_encode_frame,
     -+    .encode2        = srt_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(srt_encode_frame),
     ++    FF_CODEC_ENCODE_CB(srt_encode_frame),
           .close          = srt_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
     -@@ libavcodec/srtenc.c: const AVCodec ff_subrip_encoder = {
     -     .id             = AV_CODEC_ID_SUBRIP,
     +@@ libavcodec/srtenc.c: const FFCodec ff_subrip_encoder = {
     +     .p.id           = AV_CODEC_ID_SUBRIP,
           .priv_data_size = sizeof(SRTContext),
           .init           = srt_encode_init,
     --    .encode_sub     = srt_encode_frame,
     -+    .encode2        = srt_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(srt_encode_frame),
     ++    FF_CODEC_ENCODE_CB(srt_encode_frame),
           .close          = srt_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
     -@@ libavcodec/srtenc.c: const AVCodec ff_text_encoder = {
     -     .id             = AV_CODEC_ID_TEXT,
     +@@ libavcodec/srtenc.c: const FFCodec ff_text_encoder = {
     +     .p.id           = AV_CODEC_ID_TEXT,
           .priv_data_size = sizeof(SRTContext),
           .init           = srt_encode_init,
     --    .encode_sub     = text_encode_frame,
     -+    .encode2        = text_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(text_encode_frame),
     ++    FF_CODEC_ENCODE_CB(text_encode_frame),
           .close          = srt_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
      
       ## libavcodec/tests/avcodec.c ##
      @@ libavcodec/tests/avcodec.c: int main(void){
     -             continue;
     +             is_decoder = 1;
     +             break;
     +         case FF_CODEC_CB_TYPE_ENCODE:
     +-        case FF_CODEC_CB_TYPE_ENCODE_SUB:
     +         case FF_CODEC_CB_TYPE_RECEIVE_PACKET:
     +             is_encoder = 1;
     +             break;
     +@@ libavcodec/tests/avcodec.c: int main(void){
     + #define CHECK(TYPE, type) (codec2->cb_type == FF_CODEC_CB_TYPE_ ## TYPE && !codec2->cb.type)
     +         if (CHECK(DECODE, decode) || CHECK(DECODE_SUB, decode_sub) ||
     +             CHECK(RECEIVE_PACKET, receive_packet) ||
     +-            CHECK(ENCODE, encode) || CHECK(ENCODE_SUB, encode_sub) ||
     ++            CHECK(ENCODE, encode) ||
     +             CHECK(RECEIVE_FRAME, receive_frame)) {
     +             ERR_EXT("Codec %s does not implement its %s callback.\n",
     +                     is_decoder ? "decoding" : "encoding");
               }
     + #undef CHECK
               if (is_encoder) {
     --            if (codec->type == AVMEDIA_TYPE_SUBTITLE ^ !!codec->encode_sub)
     +-            if ((codec->type == AVMEDIA_TYPE_SUBTITLE) != (codec2->cb_type == FF_CODEC_CB_TYPE_ENCODE_SUB))
      -                ERR("Encoder %s is both subtitle encoder and not subtitle encoder.");
     -             if (!!codec->encode_sub + !!codec->encode2 + !!codec->receive_packet != 1)
     -                 ERR("Encoder %s does not implement exactly one encode API.\n");
     -             if (codec->update_thread_context || codec->update_thread_context_for_user || codec->bsfs)
     +             if (codec2->update_thread_context || codec2->update_thread_context_for_user || codec2->bsfs)
     +                 ERR("Encoder %s has decoder-only thread functions or bsf.\n");
     +             if (codec->type == AVMEDIA_TYPE_AUDIO) {
      
       ## libavcodec/ttmlenc.c ##
      @@
     @@ libavcodec/ttmlenc.c: static av_cold int ttml_encode_init(AVCodecContext *avctx)
           }
       
           return 0;
     -@@ libavcodec/ttmlenc.c: const AVCodec ff_ttml_encoder = {
     -     .id             = AV_CODEC_ID_TTML,
     +@@ libavcodec/ttmlenc.c: const FFCodec ff_ttml_encoder = {
     +     .p.id           = AV_CODEC_ID_TTML,
           .priv_data_size = sizeof(TTMLContext),
           .init           = ttml_encode_init,
     --    .encode_sub     = ttml_encode_frame,
     -+    .encode2        = ttml_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(ttml_encode_frame),
     ++    FF_CODEC_ENCODE_CB(ttml_encode_frame),
           .close          = ttml_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP,
       };
      
     + ## libavcodec/utils.c ##
     +@@ libavcodec/utils.c: int av_codec_is_encoder(const AVCodec *avcodec)
     + {
     +     const FFCodec *const codec = ffcodec(avcodec);
     +     return codec && (codec->cb_type == FF_CODEC_CB_TYPE_ENCODE     ||
     +-                     codec->cb_type == FF_CODEC_CB_TYPE_ENCODE_SUB ||
     +                      codec->cb_type == FF_CODEC_CB_TYPE_RECEIVE_PACKET);
     + }
     + 
     +
       ## libavcodec/webvttenc.c ##
      @@
       
     @@ libavcodec/webvttenc.c
      +#include "encode.h"
       #include "libavutil/avstring.h"
       #include "libavutil/bprint.h"
     - #include "libavutil/ass_split_internal.h"
     + #include "codec_internal.h"
      @@
       typedef struct {
           AVCodecContext *avctx;
     @@ libavcodec/webvttenc.c: static const ASSCodesCallbacks webvtt_callbacks = {
      +
      +    // The frame has content, so we need to set up a context
      +    if (frame->subtitle_header && frame->subtitle_header->size > 0) {
     -+        avpriv_ass_split_free(s->ass_ctx);
      +        const char* subtitle_header = (char*)frame->subtitle_header->data;
     ++        avpriv_ass_split_free(s->ass_ctx);
      +        s->ass_ctx = avpriv_ass_split(subtitle_header);
      +        s->is_default_ass_context = 0;
      +    }
     @@ libavcodec/webvttenc.c: static av_cold int webvtt_encode_init(AVCodecContext *av
      +    return 0;
       }
       
     - const AVCodec ff_webvtt_encoder = {
     -@@ libavcodec/webvttenc.c: const AVCodec ff_webvtt_encoder = {
     -     .id             = AV_CODEC_ID_WEBVTT,
     + const FFCodec ff_webvtt_encoder = {
     +@@ libavcodec/webvttenc.c: const FFCodec ff_webvtt_encoder = {
     +     .p.id           = AV_CODEC_ID_WEBVTT,
           .priv_data_size = sizeof(WebVTTContext),
           .init           = webvtt_encode_init,
     --    .encode_sub     = webvtt_encode_frame,
     -+    .encode2        = webvtt_encode_frame,
     +-    FF_CODEC_ENCODE_SUB_CB(webvtt_encode_frame),
     ++    FF_CODEC_ENCODE_CB(webvtt_encode_frame),
           .close          = webvtt_encode_close,
           .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
       };
      
       ## libavcodec/xsubenc.c ##
      @@
     - 
       #include "avcodec.h"
       #include "bytestream.h"
     + #include "codec_internal.h"
      +#include "encode.h"
     - #include "internal.h"
       #include "put_bits.h"
       
     + /**
      @@ libavcodec/xsubenc.c: static int make_tc(uint64_t ms, int *tc)
           return ms > 99;
       }
     @@ libavcodec/xsubenc.c: static int xsub_encode(AVCodecContext *avctx, unsigned cha
       }
       
       static av_cold int xsub_encoder_init(AVCodecContext *avctx)
     -@@ libavcodec/xsubenc.c: const AVCodec ff_xsub_encoder = {
     -     .type       = AVMEDIA_TYPE_SUBTITLE,
     -     .id         = AV_CODEC_ID_XSUB,
     +@@ libavcodec/xsubenc.c: const FFCodec ff_xsub_encoder = {
     +     .p.type     = AVMEDIA_TYPE_SUBTITLE,
     +     .p.id       = AV_CODEC_ID_XSUB,
           .init       = xsub_encoder_init,
     --    .encode_sub = xsub_encode,
     -+    .encode2    = xsub_encode,
     +-    FF_CODEC_ENCODE_SUB_CB(xsub_encode),
     ++    FF_CODEC_ENCODE_CB(xsub_encode),
           .caps_internal = FF_CODEC_CAP_INIT_THREADSAFE,
       };
 21:  314d1da505 ! 22:  fa0b5c2077 fftools/ffmpeg: Introduce subtitle filtering and new frame-based subtitle encoding
     @@ Commit message
          Justification for changed test refs:
      
          - sub2video
     -      The new results are identical excepting the last frame which
     -      is due to the implementation changes
     +      The previous results were incorrect. The command line for this test
     +      specifies -r 5 (5 frames per second), which is now fulfilled by the
     +      additional frames in the reference output.
     +      Example: The first subtitle time is 499000, the second is 15355000,
     +      which means 0.5s and 15.35s with a difference of 14.85s.
     +      15s * 5fps = 75 frames and that's now the exact number of video
     +      frames between these two subtitle events.
      
          - sub2video_basic
            The previous results had some incorrect output because multiple
     @@ Commit message
            CRC is due to the different blending algorithm that is being used.
      
          - sub2video_time_limited
     -      The third frame in the previous ref was a repetition, which doesn't
     -      happen anymore with the new subtitle filtering.
     +      Subtitle frames are emitted to the filter graphs at a 5 fps rate
     +      by default. The time limit for this test is 15s * 5fps = 75 frames
     +      which matches the count in the new ref.
      
          - sub-dvb
            Running ffprobe -show_frames on the source file shows that there
     @@ Commit message
          Signed-off-by: softworkz <softworkz@hotmail.com>
      
       ## fftools/ffmpeg.c ##
     -@@ fftools/ffmpeg.c: static int want_sdp = 1;
     +@@ fftools/ffmpeg.c: int want_sdp = 1;
       static BenchmarkTimeStamps current_time;
       AVIOContext *progress_avio = NULL;
       
     @@ fftools/ffmpeg.c: static void ffmpeg_cleanup(int ret)
                   InputFilter *ifilter = fg->inputs[j];
      -            struct InputStream *ist = ifilter->ist;
       
     -             while (av_fifo_size(ifilter->frame_queue)) {
     +             if (ifilter->frame_queue) {
                       AVFrame *frame;
      @@ fftools/ffmpeg.c: static void ffmpeg_cleanup(int ret)
     +                 av_fifo_freep2(&ifilter->frame_queue);
                   }
     -             av_fifo_freep(&ifilter->frame_queue);
                   av_freep(&ifilter->displaymatrix);
      -            if (ist->sub2video.sub_queue) {
     --                while (av_fifo_size(ist->sub2video.sub_queue)) {
     --                    AVSubtitle sub;
     --                    av_fifo_generic_read(ist->sub2video.sub_queue,
     --                                         &sub, sizeof(sub), NULL);
     +-                AVSubtitle sub;
     +-                while (av_fifo_read(ist->sub2video.sub_queue, &sub, 1) >= 0)
      -                    avsubtitle_free(&sub);
     --                }
     --                av_fifo_freep(&ist->sub2video.sub_queue);
     +-                av_fifo_freep2(&ist->sub2video.sub_queue);
      -            }
                   av_buffer_unref(&ifilter->hw_frames_ctx);
                   av_freep(&ifilter->name);
     @@ fftools/ffmpeg.c: static void ffmpeg_cleanup(int ret)
      +    ////av_freep(&subtitle_out);
       
           /* close files */
     -     for (i = 0; i < nb_output_files; i++) {
     +     for (i = 0; i < nb_output_files; i++)
      @@ fftools/ffmpeg.c: static void ffmpeg_cleanup(int ret)
     +     for (i = 0; i < nb_input_streams; i++) {
     +         InputStream *ist = input_streams[i];
     + 
     ++        if (ist->prev_sub.subtitle == ist->decoded_frame)
     ++            // Prevent double-free
     ++            ist->prev_sub.subtitle = NULL;
     ++
               av_frame_free(&ist->decoded_frame);
               av_packet_free(&ist->pkt);
               av_dict_free(&ist->decoder_opts);
     @@ fftools/ffmpeg.c: static void ffmpeg_cleanup(int ret)
               avcodec_free_context(&ist->dec_ctx);
       
               av_freep(&input_streams[i]);
     -@@ fftools/ffmpeg.c: error:
     -     exit_program(1);
     +@@ fftools/ffmpeg.c: static void do_audio_out(OutputFile *of, OutputStream *ost,
     +         exit_program(1);
       }
       
      -static void do_subtitle_out(OutputFile *of,
      -                            OutputStream *ost,
      -                            AVSubtitle *sub)
      +static void encode_subtitle_frame(OutputFile *of, OutputStream *ost, AVFrame *frame, AVPacket *pkt, int64_t pts_offset)
     - {
     --    int subtitle_out_max_size = 1024 * 1024;
     --    int subtitle_out_size, nb, i;
     ++{
      +    AVCodecContext *enc = ost->enc_ctx;
      +    int ret;
      +
     @@ fftools/ffmpeg.c: error:
      +}
      +
      +static void do_subtitle_out(OutputFile *of, OutputStream *ost, AVFrame *frame)
     -+{
     + {
     +-    int subtitle_out_max_size = 1024 * 1024;
     +-    int subtitle_out_size, nb, i;
      +    int nb, i;
           AVCodecContext *enc;
           AVPacket *pkt = ost->pkt;
     @@ fftools/ffmpeg.c: error:
      +        return;
           }
       
     ++    ////if (ost->last_subtitle_pts && ost->last_subtitle_pts == frame->subtitle_timing.start_pts) {
     ++    ////    av_log(NULL, AV_LOG_DEBUG, "Ignoring subtitle frame with duplicate subtitle_pts\n");
     ++    ////    return;
     ++    ////}
     ++
     ++    ost->last_subtitle_pts = frame->subtitle_timing.start_pts;
     ++
      +    init_output_stream_wrapper(ost, frame, 1);
      +
      +    enc = ost->enc_ctx;
     @@ fftools/ffmpeg.c: static int ifilter_send_frame(InputFilter *ifilter, AVFrame *f
           }
       
      @@ fftools/ffmpeg.c: static int ifilter_send_eof(InputFilter *ifilter, int64_t pts)
     +             return ret;
     +     } else {
               // the filtergraph was never configured
     -         if (ifilter->format < 0)
     -             ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar);
     +-        if (ifilter->format < 0) {
     +-            ret = ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar);
     +-            if (ret < 0)
     +-                return ret;
     +-        }
      -        if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO)) {
     ++        if (ifilter->format < 0)
     ++            ifilter_parameters_from_codecpar(ifilter, ifilter->ist->st->codecpar);
      +        if (ifilter->format < 0 && (ifilter->type == AVMEDIA_TYPE_AUDIO || ifilter->type == AVMEDIA_TYPE_VIDEO || ifilter->type == AVMEDIA_TYPE_SUBTITLE)) {
                   av_log(NULL, AV_LOG_ERROR, "Cannot determine format of input stream %d:%d after EOF\n", ifilter->ist->file_index, ifilter->ist->st->index);
                   return AVERROR_INVALIDDATA;
     @@ fftools/ffmpeg.c: fail:
       }
       
      -static int transcode_subtitles(InputStream *ist, AVPacket *pkt, int *got_output,
     ++static void subtitle_send_kickoff(InputStream *ist, int64_t heartbeat_pts)
     ++{
     ++    AVFrame *frame;
     ++    int64_t pts;
     ++
     ++    frame = av_frame_alloc();
     ++    if (!frame) {
     ++        av_log(ist->dec_ctx, AV_LOG_ERROR, "Unable to alloc frame (out of memory).\n");
     ++        return;
     ++    }
     ++
     ++    frame->type = AVMEDIA_TYPE_SUBTITLE;
     ++    av_frame_get_buffer2(frame, 0);
     ++
     ++    frame->format = (uint16_t)ist->dec_ctx->subtitle_type;
     ++
     ++    pts = FFMAX(heartbeat_pts, ist->subtitle_kickoff.last_pts) + av_rescale_q(10, (AVRational){ 1, 1000 }, ist->st->time_base);
     ++
     ++    frame->width = ist->subtitle_kickoff.w;
     ++    frame->height = ist->subtitle_kickoff.h;
     ++    frame->subtitle_timing.start_pts = av_rescale_q(pts, ist->st->time_base, AV_TIME_BASE_Q);
     ++    frame->subtitle_timing.duration =  av_rescale_q(10, (AVRational){ 1, 1000 }, AV_TIME_BASE_Q);
     ++    frame->pts = pts;
     ++
     ++    av_log(NULL, AV_LOG_WARNING, "subtitle_kickoff: call subtitle_resend_current %"PRId64" frame->format: %d\n", pts, frame->format);
     ++
     ++    ist->subtitle_kickoff.last_pts = pts;
     ++
     ++    send_frame_to_filters(ist, frame);
     ++
     ++    av_frame_free(&frame);
     ++}
     ++
     ++// Opposed to the earlier "subtitle hearbeat", this is primarily aimed at
     ++// sending an initial subtitle frame to the filters for propagating the initial
     ++// timing values and to avoid that too much time is spent on a single "HW" decoder
     ++// which could overflow its buffer pool otherwise
     ++static void subtitle_kickoff(InputStream *ist, int64_t pts)
     ++{
     ++    int i;
     ++    int64_t pts2;
     ++    int64_t diff;
     ++
     ++    if (ist->st->codecpar->codec_type != AVMEDIA_TYPE_VIDEO)
     ++        return;
     ++
     ++    for (i = 0; i < nb_input_streams; i++) {
     ++        InputStream *ist2 = input_streams[i];
     ++        unsigned j, nb_reqs;
     ++
     ++        if (!ist2->subtitle_kickoff.is_active)
     ++            continue;
     ++        /* It's primarily the initial send that matters and which propagates
     ++         * the right start times to subtitle filters depending on input from decoding */
     ++        diff = av_rescale_q(5000, (AVRational){ 1, 1000 }, ist2->st->time_base);
     ++        pts2 = av_rescale_q(pts, ist->st->time_base, ist2->st->time_base);
     ++
     ++        /* do not send a kickoff frame if the subtitle is already ahead */
     ++        if (pts2 - diff <= ist2->subtitle_kickoff.last_pts)
     ++            continue;
     ++
     ++        for (j = 0, nb_reqs = 0; j < ist2->nb_filters; j++) {
     ++            if (ist2->filters[j]->filter == NULL) {
     ++                subtitle_send_kickoff(ist2, pts2);
     ++                return;
     ++            }
     ++            nb_reqs += av_buffersrc_get_nb_failed_requests(ist2->filters[j]->filter);
     ++        }
     ++        if (nb_reqs) {
     ++            av_log(NULL, AV_LOG_WARNING, "subtitle_kickoff: resend - pts: %"PRIi64"\n", pts2);
     ++            subtitle_send_kickoff(ist2, pts2);
     ++        }
     ++    }
     ++}
     ++
      +static InputStream *get_input_stream(OutputStream *ost)
      +{
      +    if (ost->source_index >= 0)
     @@ fftools/ffmpeg.c: fail:
      +    AVFrame *decoded_frame;
      +    AVCodecContext *avctx = ist->dec_ctx;
      +    int i = 0, ret = 0, err = 0;
     ++    int64_t pts;
      +
     -+    if (!ist->decoded_frame && !(ist->decoded_frame = av_frame_alloc()))
     ++    if (!ist->decoded_frame && !((ist->decoded_frame = av_frame_alloc())))
      +        return AVERROR(ENOMEM);
      +    decoded_frame = ist->decoded_frame;
      +    decoded_frame->type = AVMEDIA_TYPE_SUBTITLE;
     -+    decoded_frame->format = avcodec_descriptor_get_subtitle_format(avctx->codec_descriptor);
     - 
     --    check_decode_result(NULL, got_output, ret);
     ++    decoded_frame->format = (uint16_t)avctx->subtitle_type;
     ++
      +    if (!ist->subtitle_header && avctx->subtitle_header && avctx->subtitle_header_size > 0) {
      +        ist->subtitle_header = av_buffer_allocz(avctx->subtitle_header_size + 1);
      +        if (!ist->subtitle_header)
     @@ fftools/ffmpeg.c: fail:
      +    }
      +
      +    ret = decode(avctx, decoded_frame, got_output, pkt);
     -+
     + 
     +-    check_decode_result(NULL, got_output, ret);
      +    if (ret != AVERROR_EOF)
      +        check_decode_result(NULL, got_output, ret);
       
     @@ fftools/ffmpeg.c: fail:
      +        if (!pkt->size) {
      +            // Flush
      +            for (i = 0; i < ist->nb_filters; i++) {
     -+                ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL);
     -+                if (ret != AVERROR_EOF && ret < 0)
     -+                    av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n");
     ++                if (ist->filters[i]->filter != NULL) {
     ++                    ret = av_buffersrc_add_frame(ist->filters[i]->filter, NULL);
     ++                    if (ret != AVERROR_EOF && ret < 0)
     ++                        av_log(NULL, AV_LOG_WARNING, "Flush the frame error.\n");
     ++                }
      +            }
      +        }
               return ret;
     @@ fftools/ffmpeg.c: fail:
      -        FFSWAP(int,        *got_output, ist->prev_sub.got_output);
      -        FFSWAP(int,        ret,         ist->prev_sub.ret);
      -        FFSWAP(AVSubtitle, subtitle,    ist->prev_sub.subtitle);
     -+        FFSWAP(int,        *got_output,   ist->prev_sub.got_output);
     -+        FFSWAP(int,        ret,           ist->prev_sub.ret);
     -+        FFSWAP(AVFrame*,   decoded_frame, ist->prev_sub.subtitle);
     ++        FFSWAP(int,        *got_output,        ist->prev_sub.got_output);
     ++        FFSWAP(int,        ret,                ist->prev_sub.ret);
     ++        FFSWAP(AVFrame*,   ist->decoded_frame, ist->prev_sub.subtitle);
     ++        decoded_frame = ist->decoded_frame;
               if (end <= 0)
      -            goto out;
      +            return (int)end;
     @@ fftools/ffmpeg.c: fail:
      -        sub2video_update(ist, INT64_MIN, &subtitle);
      -    } else if (ist->nb_filters) {
      -        if (!ist->sub2video.sub_queue)
     --            ist->sub2video.sub_queue = av_fifo_alloc(8 * sizeof(AVSubtitle));
     +-            ist->sub2video.sub_queue = av_fifo_alloc2(8, sizeof(AVSubtitle), AV_FIFO_FLAG_AUTO_GROW);
      -        if (!ist->sub2video.sub_queue)
      -            exit_program(1);
     --        if (!av_fifo_space(ist->sub2video.sub_queue)) {
     --            ret = av_fifo_realloc2(ist->sub2video.sub_queue, 2 * av_fifo_size(ist->sub2video.sub_queue));
     --            if (ret < 0)
     --                exit_program(1);
      +    decoded_frame->type = AVMEDIA_TYPE_SUBTITLE;
     -+    decoded_frame->format = avcodec_descriptor_get_subtitle_format(avctx->codec_descriptor);
     + 
     +-        ret = av_fifo_write(ist->sub2video.sub_queue, &subtitle, 1);
     +-        if (ret < 0)
     +-            exit_program(1);
     +-        free_sub = 0;
     +-    }
     ++    if (decoded_frame->format == AV_SUBTITLE_FMT_UNKNOWN)
     ++        decoded_frame->format = (uint16_t)avctx->subtitle_type;
      +
      +    if ((ret = av_buffer_replace(&decoded_frame->subtitle_header, ist->subtitle_header)) < 0)
      +        return ret;
     -+
     + 
     +-    if (!subtitle.num_rects)
     +-        goto out;
      +    decoded_frame->pts = av_rescale_q(decoded_frame->subtitle_timing.start_pts, AV_TIME_BASE_Q, ist->st->time_base);
     -+
     + 
     +-    ist->frames_decoded++;
     ++    pts     = av_rescale_q(decoded_frame->subtitle_timing.start_pts,
     ++                           AV_TIME_BASE_Q, ist->st->time_base);
     + 
     +-    for (i = 0; i < nb_output_streams; i++) {
     +-        OutputStream *ost = output_streams[i];
     ++    ist->subtitle_kickoff.last_pts = decoded_frame->pts = pts;
     + 
     +-        if (!check_output_constraints(ist, ost) || !ost->encoding_needed
     +-            || ost->enc->type != AVMEDIA_TYPE_SUBTITLE)
     +-            continue;
      +    if (ist->nb_filters > 0) {
      +        AVFrame *filter_frame = av_frame_clone(decoded_frame);
      +        if (!filter_frame) {
      +            err = AVERROR(ENOMEM);
      +            goto end;
     -         }
     --        av_fifo_generic_write(ist->sub2video.sub_queue, &subtitle, sizeof(subtitle), NULL);
     --        free_sub = 0;
     --    }
     ++        }
       
     --    if (!subtitle.num_rects)
     --        goto out;
     +-        do_subtitle_out(output_files[ost->file_index], ost, &subtitle);
      +        err = send_frame_to_filters(ist, filter_frame);
      +        av_frame_free(&filter_frame);
     -+    }
     +     }
       
     --    ist->frames_decoded++;
     +-out:
     +-    if (free_sub)
     +-        avsubtitle_free(&subtitle);
     +-    return ret;
      +    if (err >= 0) {
      +        for (i = 0; i < nb_output_streams; i++) {
      +            OutputStream *ost = output_streams[i];
      +            InputStream *ist_src = get_input_stream(ost);
     - 
     --    for (i = 0; i < nb_output_streams; i++) {
     --        OutputStream *ost = output_streams[i];
     ++
      +            if (!ist_src || !check_output_constraints(ist, ost)
      +                || ist_src != ist
      +                || !ost->encoding_needed
      +                || ost->enc->type != AVMEDIA_TYPE_SUBTITLE)
      +                continue;
     - 
     --        if (!check_output_constraints(ist, ost) || !ost->encoding_needed
     --            || ost->enc->type != AVMEDIA_TYPE_SUBTITLE)
     --            continue;
     ++
      +            if (ost->filter && ost->filter->filter->nb_inputs > 0)
      +                continue;
     - 
     --        do_subtitle_out(output_files[ost->file_index], ost, &subtitle);
     ++
      +            do_subtitle_out(output_files[ost->file_index], ost, decoded_frame);
      +        }
     -     }
     - 
     --out:
     --    if (free_sub)
     --        avsubtitle_free(&subtitle);
     --    return ret;
     ++    }
     ++
      +end:
      +    av_frame_unref(decoded_frame);
      +    return err < 0 ? err : ret;
     @@ fftools/ffmpeg.c: static int process_input_packet(InputStream *ist, const AVPack
                   if (!pkt && ret >= 0)
                       ret = AVERROR_EOF;
                   av_packet_unref(avpkt);
     -@@ fftools/ffmpeg.c: FF_ENABLE_DEPRECATION_WARNINGS
     +@@ fftools/ffmpeg.c: static int init_input_stream(int ist_index, char *error, int error_len)
           return 0;
       }
       
     @@ fftools/ffmpeg.c: static int init_output_stream_encode(OutputStream *ost, AVFram
                   enc_ctx->width     = input_streams[ost->source_index]->st->codecpar->width;
                   enc_ctx->height    = input_streams[ost->source_index]->st->codecpar->height;
               }
     +@@ fftools/ffmpeg.c: static int init_output_stream_encode(OutputStream *ost, AVFrame *frame)
     +     return 0;
     + }
     + 
     ++static enum AVSubtitleType get_subtitle_format(const AVCodecDescriptor *codec_descriptor)
     ++{
     ++    if(codec_descriptor->props & AV_CODEC_PROP_BITMAP_SUB)
     ++        return AV_SUBTITLE_FMT_BITMAP;
     ++
     ++    if(codec_descriptor->props & AV_CODEC_PROP_TEXT_SUB)
     ++        return AV_SUBTITLE_FMT_ASS;
     ++
     ++    return AV_SUBTITLE_FMT_UNKNOWN;
     ++}
     ++
     + static int init_output_stream(OutputStream *ost, AVFrame *frame,
     +                               char *error, int error_len)
     + {
      @@ fftools/ffmpeg.c: static int init_output_stream(OutputStream *ost, AVFrame *frame,
               }
       
     @@ fftools/ffmpeg.c: static int init_output_stream(OutputStream *ost, AVFrame *fram
      -                snprintf(error, error_len,
      -                         "Subtitle encoding currently only possible from text to text "
      -                         "or bitmap to bitmap");
     -+            AVCodecDescriptor const *input_descriptor     = avcodec_descriptor_get(dec->codec_id);
      +            AVCodecDescriptor const *output_descriptor    = avcodec_descriptor_get(ost->enc_ctx->codec_id);
     -+            const enum AVSubtitleType in_subtitle_format  = output_descriptor ? avcodec_descriptor_get_subtitle_format(input_descriptor) : AV_SUBTITLE_FMT_UNKNOWN;
     -+            const enum AVSubtitleType out_subtitle_format = output_descriptor ? avcodec_descriptor_get_subtitle_format(output_descriptor) : AV_SUBTITLE_FMT_UNKNOWN;
     ++            const enum AVSubtitleType in_subtitle_format  = (uint16_t)dec->subtitle_type;
     ++            const enum AVSubtitleType out_subtitle_format = output_descriptor ? get_subtitle_format(output_descriptor) : AV_SUBTITLE_FMT_UNKNOWN;
      +
      +            if (ist->nb_filters == 0 && in_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN && out_subtitle_format != AV_SUBTITLE_FMT_UNKNOWN
      +                && in_subtitle_format != out_subtitle_format) {
     @@ fftools/ffmpeg.c: static int init_output_stream(OutputStream *ost, AVFrame *fram
                       return AVERROR_INVALIDDATA;
                   }
               }
     -@@ fftools/ffmpeg.c: static int transcode_init(void)
     -     for (i = 0; i < nb_output_streams; i++) {
     -         if (!output_streams[i]->stream_copy &&
     -             (output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_VIDEO ||
     --             output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_AUDIO))
     -+             output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_AUDIO ||
     -+             output_streams[i]->enc_ctx->codec_type == AVMEDIA_TYPE_SUBTITLE))
     -             continue;
     +@@ fftools/ffmpeg.c: static OutputStream *choose_output(void)
     +                        av_rescale_q(ost->last_mux_dts, ost->st->time_base,
     +                                     AV_TIME_BASE_Q);
     +         if (ost->last_mux_dts == AV_NOPTS_VALUE)
     +-            av_log(NULL, AV_LOG_DEBUG,
     ++            av_log(NULL, AV_LOG_INFO,
     +                 "cur_dts is invalid st:%d (%d) [init:%d i_done:%d finish:%d] (this is harmless if it occurs once at the start per stream)\n",
     +                 ost->st->index, ost->st->id, ost->initialized, ost->inputs_done, ost->finished);
       
     -         ret = init_output_stream_wrapper(output_streams[i], NULL, 0);
      @@ fftools/ffmpeg.c: static int process_input(int file_index)
                      av_ts2timestr(input_files[ist->file_index]->ts_offset, &AV_TIME_BASE_Q));
           }
       
      -    sub2video_heartbeat(ist, pkt->pts);
     --
     ++    subtitle_kickoff(ist, pkt->pts);
     + 
           process_input_packet(ist, pkt, 0);
       
     - discard_packet:
     +@@ fftools/ffmpeg.c: static int transcode(void)
     +     /* at the end of stream, we must flush the decoder buffers */
     +     for (i = 0; i < nb_input_streams; i++) {
     +         ist = input_streams[i];
     ++        ist->subtitle_kickoff.is_active = 0;
     +         if (!input_files[ist->file_index]->eof_reached) {
     +             process_input_packet(ist, NULL, 0);
     +         }
      
       ## fftools/ffmpeg.h ##
      @@ fftools/ffmpeg.h: typedef struct InputStream {
     @@ fftools/ffmpeg.h: typedef struct InputStream {
           } prev_sub;
       
      -    struct sub2video {
     --        int64_t last_pts;
     ++    struct subtitle_kickoff {
     ++        int is_active;
     +         int64_t last_pts;
      -        int64_t end_pts;
     --        AVFifoBuffer *sub_queue;    ///< queue of AVSubtitle* before filter init
     +-        AVFifo *sub_queue;    ///< queue of AVSubtitle* before filter init
      -        AVFrame *frame;
     --        int w, h;
     +         int w, h;
      -        unsigned int initialize; ///< marks if sub2video_update should force an initialization
      -    } sub2video;
     ++    } subtitle_kickoff;
     ++
      +    AVBufferRef *subtitle_header;
       
           /* decoded data from this stream goes into all those filters
            * currently video and audio only */
     +@@ fftools/ffmpeg.h: typedef struct OutputStream {
     +     int64_t first_pts;
     +     /* dts of the last packet sent to the muxer */
     +     int64_t last_mux_dts;
     ++    /* subtitle_pts values of the last subtitle frame having arrived for encoding */
     ++    int64_t last_subtitle_pts;
     +     // the timebase of the packets sent to the muxer
     +     AVRational mux_timebase;
     +     AVRational enc_timebase;
      @@ fftools/ffmpeg.h: int filtergraph_is_simple(FilterGraph *fg);
       int init_simple_filtergraph(InputStream *ist, OutputStream *ost);
       int init_complex_filtergraph(FilterGraph *fg);
     @@ fftools/ffmpeg_filter.c: static void init_input_filter(FilterGraph *fg, AVFilter
                       continue;
                   if (check_stream_specifier(s, s->streams[i], *p == ':' ? p + 1 : p) == 1) {
                       st = s->streams[i];
     -@@ fftools/ffmpeg_filter.c: static void init_input_filter(FilterGraph *fg, AVFilterInOut *in)
     -     ifilter->type   = ist->st->codecpar->codec_type;
     -     ifilter->name   = describe_filter_link(fg, in, 1);
     - 
     -+    if (ist->st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE && ist->dec_ctx) {
     -+        const AVCodecDescriptor *codec_descriptor = ist->dec_ctx->codec_descriptor;
     -+        if (!codec_descriptor)
     -+            codec_descriptor = avcodec_descriptor_get(ist->dec_ctx->codec_id);
     -+
     -+        // For subtitles, we need to set the format here. Would we leave the format
     -+        // at -1, it would delay processing (ifilter_has_all_input_formats()) until
     -+        // the first subtile frame arrives, which could never happen in the worst case
     -+        fg->inputs[fg->nb_inputs - 1]->format = avcodec_descriptor_get_subtitle_format(codec_descriptor);
     -+    }
     -+
     -     ifilter->frame_queue = av_fifo_alloc(8 * sizeof(AVFrame*));
     -     if (!ifilter->frame_queue)
     -         exit_program(1);
      @@ fftools/ffmpeg_filter.c: static int insert_filter(AVFilterContext **last_filter, int *pad_idx,
           return 0;
       }
     @@ fftools/ffmpeg_filter.c: void check_filter_outputs(void)
      +        ret = AVERROR(EINVAL);
      +        goto fail;
      +    }
     ++
     ++    if (ist->st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE && ist->dec_ctx) {
     ++        // For subtitles, we need to set the format here. Would we leave the format
     ++        // at -1, it would delay processing (ifilter_has_all_input_formats()) until
     ++        // the first subtile frame arrives, which could never happen in the worst case
     ++        ifilter->format = (uint16_t)ist->dec_ctx->subtitle_type;
     ++    }
     ++
     ++    ist->subtitle_kickoff.is_active = 1;
       
      -    /* Compute the size of the canvas for the subtitles stream.
      -       If the subtitles codecpar has set a size, use it. Otherwise use the
     @@ fftools/ffmpeg_filter.c: void check_filter_outputs(void)
      -        av_log(avf, AV_LOG_INFO, "sub2video: using %dx%d canvas\n", w, h);
      +        w = ist->dec_ctx->width;
      +        h = ist->dec_ctx->height;
     -     }
     --    ist->sub2video.w = ifilter->width  = w;
     --    ist->sub2video.h = ifilter->height = h;
     - 
     --    ifilter->width  = ist->dec_ctx->width  ? ist->dec_ctx->width  : ist->sub2video.w;
     --    ifilter->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h;
     ++    }
     ++
      +    if (!(w && h) && ist->dec_ctx->subtitle_header) {
      +        ASSSplitContext *ass_ctx = avpriv_ass_split((char *)ist->dec_ctx->subtitle_header);
      +        ASS *ass = (ASS *)ass_ctx;
      +        w = ass->script_info.play_res_x;
      +        h = ass->script_info.play_res_y;
      +        avpriv_ass_split_free(ass_ctx);
     -+    }
     +     }
     +-    ist->sub2video.w = ifilter->width  = w;
     +-    ist->sub2video.h = ifilter->height = h;
     + 
     +-    ifilter->width  = ist->dec_ctx->width  ? ist->dec_ctx->width  : ist->sub2video.w;
     +-    ifilter->height = ist->dec_ctx->height ? ist->dec_ctx->height : ist->sub2video.h;
     ++    ist->subtitle_kickoff.w = w;
     ++    ist->subtitle_kickoff.h = h;
     ++    av_log(ifilter, AV_LOG_INFO, "subtitle input filter: decoding size %dx%d\n", ist->subtitle_kickoff.w, ist->subtitle_kickoff.h);
       
      -    /* rectangles are AV_PIX_FMT_PAL8, but we have no guarantee that the
      -       palettes for all rectangles are identical or compatible */
     @@ fftools/ffmpeg_filter.c: void check_filter_outputs(void)
      -        return AVERROR(ENOMEM);
      -    ist->sub2video.last_pts = INT64_MIN;
      -    ist->sub2video.end_pts  = INT64_MIN;
     ++    ist->subtitle_kickoff.last_pts = INT64_MIN;
     ++
      +    snprintf(name, sizeof(name), "graph %d subtitle input from stream %d:%d", fg->index,
      +             ist->file_index, ist->st->index);
      +
     @@ fftools/ffmpeg_filter.c: void check_filter_outputs(void)
      +    if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name,
      +                                            args.str, NULL, fg->graph)) < 0)
      +        goto fail;
     -+
     + 
     +-    /* sub2video structure has been (re-)initialized.
     +-       Mark it as such so that the system will be
     +-       initialized with the first received heartbeat. */
     +-    ist->sub2video.initialize = 1;
      +    par->hw_frames_ctx = ifilter->hw_frames_ctx;
      +    par->format = ifilter->format;
      +    par->width = ifilter->width;
     @@ fftools/ffmpeg_filter.c: void check_filter_outputs(void)
      +            if (ret < 0)
      +                return ret;
      +        }
     - 
     --    /* sub2video structure has been (re-)initialized.
     --       Mark it as such so that the system will be
     --       initialized with the first received heartbeat. */
     --    ist->sub2video.initialize = 1;
     ++
      +        av_log(NULL, AV_LOG_INFO, "Auto-inserting graphicsub2video filter\n");
      +        ret = insert_filter(&last_filter, &pad_idx, "graphicsub2video", NULL);
      +        if (ret < 0)
     @@ fftools/ffmpeg_filter.c: static int configure_input_filter(FilterGraph *fg, Inpu
           default: av_assert0(0); return 0;
           }
       }
     +@@ fftools/ffmpeg_filter.c: static void cleanup_filtergraph(FilterGraph *fg)
     + static int filter_is_buffersrc(const AVFilterContext *f)
     + {
     +     return f->nb_inputs == 0 &&
     +-           (!strcmp(f->filter->name, "buffer") ||
     +-            !strcmp(f->filter->name, "abuffer"));
     ++           (!strcmp(f->filter->name, "buffersrc") ||
     ++            !strcmp(f->filter->name, "abuffersrc") ||
     ++            !strcmp(f->filter->name, "sbuffersrc"));
     + }
     + 
     + static int graph_is_meta(AVFilterGraph *graph)
      @@ fftools/ffmpeg_filter.c: int configure_filtergraph(FilterGraph *fg)
               }
           }
     @@ fftools/ffmpeg_filter.c: int configure_filtergraph(FilterGraph *fg)
      -    for (i = 0; i < fg->nb_inputs; i++) {
      -        InputStream *ist = fg->inputs[i]->ist;
      -        if (ist->sub2video.sub_queue && ist->sub2video.frame) {
     --            while (av_fifo_size(ist->sub2video.sub_queue)) {
     --                AVSubtitle tmp;
     --                av_fifo_generic_read(ist->sub2video.sub_queue, &tmp, sizeof(tmp), NULL);
     +-            AVSubtitle tmp;
     +-            while (av_fifo_read(ist->sub2video.sub_queue, &tmp, 1) >= 0) {
      -                sub2video_update(ist, INT64_MIN, &tmp);
      -                avsubtitle_free(&tmp);
      -            }
     @@ fftools/ffmpeg_filter.c: int configure_filtergraph(FilterGraph *fg)
       
       fail:
      @@ fftools/ffmpeg_filter.c: int ifilter_parameters_from_frame(InputFilter *ifilter, const AVFrame *frame)
     -     ifilter->sample_rate         = frame->sample_rate;
     -     ifilter->channels            = frame->channels;
     -     ifilter->channel_layout      = frame->channel_layout;
     +     ifilter->width               = frame->width;
     +     ifilter->height              = frame->height;
     +     ifilter->sample_aspect_ratio = frame->sample_aspect_ratio;
      +    ifilter->type                = frame->type;
       
     -     av_freep(&ifilter->displaymatrix);
     -     sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DISPLAYMATRIX);
     +     ifilter->sample_rate         = frame->sample_rate;
     +     ret = av_channel_layout_copy(&ifilter->ch_layout, &frame->ch_layout);
      
       ## fftools/ffmpeg_hw.c ##
      @@ fftools/ffmpeg_hw.c: int hw_device_setup_for_encode(OutputStream *ost)
     @@ fftools/ffmpeg_hw.c: int hw_device_setup_for_encode(OutputStream *ost)
                   ((AVHWFramesContext*)frames_ref->data)->format ==
      
       ## fftools/ffmpeg_opt.c ##
     +@@ fftools/ffmpeg_opt.c: static void add_input_streams(OptionsContext *o, AVFormatContext *ic)
     +                 av_log(NULL, AV_LOG_FATAL, "Invalid canvas size: %s.\n", canvas_size);
     +                 exit_program(1);
     +             }
     ++            ist->subtitle_kickoff.is_active = 1;
     +             break;
     +         }
     +         case AVMEDIA_TYPE_ATTACHMENT:
      @@ fftools/ffmpeg_opt.c: static void init_output_filter(OutputFilter *ofilter, OptionsContext *o,
           switch (ofilter->type) {
           case AVMEDIA_TYPE_VIDEO: ost = new_video_stream(o, oc, -1); break;
     @@ tests/ref/fate/sub2video
      +0,        254,        254,        1,   518400, 0x7a11c812
      +0,        255,        255,        1,   518400, 0x7a11c812
      +0,        256,        256,        1,   518400, 0x7a11c812
     -+0,        257,        257,        1,   518400, 0x7a11c812
     ++0,        257,        257,        1,   518400, 0x34cdddee
       1,   51433000,   51433000,  2366000,     3059, 0xc95b8a05
       0,        258,        258,        1,   518400, 0x34cdddee
      -0,        269,        269,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        280,        280,        1,   518400, 0x4db4ce51
      +0,        281,        281,        1,   518400, 0x4db4ce51
      +0,        282,        282,        1,   518400, 0x4db4ce51
     -+0,        283,        283,        1,   518400, 0x4db4ce51
     ++0,        283,        283,        1,   518400, 0xe6bc0ea9
       1,   56663000,   56663000,  1262000,     1013, 0xc9ae89b7
       0,        284,        284,        1,   518400, 0xe6bc0ea9
      -0,        290,        290,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        287,        287,        1,   518400, 0xe6bc0ea9
      +0,        288,        288,        1,   518400, 0xe6bc0ea9
      +0,        289,        289,        1,   518400, 0xe6bc0ea9
     -+0,        290,        290,        1,   518400, 0xe6bc0ea9
     ++0,        290,        290,        1,   518400, 0xa8643af7
       1,   58014000,   58014000,  1661000,      969, 0xe01878f0
       0,        291,        291,        1,   518400, 0xa8643af7
      -0,        298,        298,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        351,        351,        1,   518400, 0x378e3fd0
      +0,        352,        352,        1,   518400, 0x378e3fd0
      +0,        353,        353,        1,   518400, 0x378e3fd0
     -+0,        354,        354,        1,   518400, 0x378e3fd0
     ++0,        354,        354,        1,   518400, 0xa3782469
       1,   70819000,   70819000,  1865000,     1709, 0xb4d5a1bd
       0,        355,        355,        1,   518400, 0xa3782469
      -0,        363,        363,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        371,        371,        1,   518400, 0xba23a0d5
      +0,        372,        372,        1,   518400, 0xba23a0d5
      +0,        373,        373,        1,   518400, 0xba23a0d5
     -+0,        374,        374,        1,   518400, 0xba23a0d5
     ++0,        374,        374,        1,   518400, 0x129de2f8
       1,   74806000,   74806000,  1831000,     2116, 0x96514097
       0,        375,        375,        1,   518400, 0x129de2f8
      -0,        383,        383,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        387,        387,        1,   518400, 0x19772f0f
      +0,        388,        388,        1,   518400, 0x19772f0f
      +0,        389,        389,        1,   518400, 0x19772f0f
     -+0,        390,        390,        1,   518400, 0x19772f0f
     ++0,        390,        390,        1,   518400, 0x56f54e73
       1,   78051000,   78051000,  1524000,      987, 0x7b927a27
       0,        391,        391,        1,   518400, 0x56f54e73
      -0,        398,        398,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        395,        395,        1,   518400, 0x56f54e73
      +0,        396,        396,        1,   518400, 0x56f54e73
      +0,        397,        397,        1,   518400, 0x56f54e73
     -+0,        398,        398,        1,   518400, 0x56f54e73
     ++0,        398,        398,        1,   518400, 0x300b5247
       1,   79644000,   79644000,  2662000,     2956, 0x190778f7
       0,        399,        399,        1,   518400, 0x300b5247
      +0,        400,        400,        1,   518400, 0x300b5247
     @@ tests/ref/fate/sub2video
      +0,        411,        411,        1,   518400, 0x300b5247
       1,   82380000,   82380000,  2764000,     3094, 0xc021b7d3
      -0,        412,        412,        1,   518400, 0xbab197ea
     -+0,        412,        412,        1,   518400, 0x300b5247
     ++0,        412,        412,        1,   518400, 0x6fd028fa
       0,        413,        413,        1,   518400, 0x6fd028fa
      -0,        426,        426,        1,   518400, 0xbab197ea
      +0,        414,        414,        1,   518400, 0x6fd028fa
     @@ tests/ref/fate/sub2video
      +0,        423,        423,        1,   518400, 0x6fd028fa
      +0,        424,        424,        1,   518400, 0x6fd028fa
      +0,        425,        425,        1,   518400, 0x6fd028fa
     -+0,        426,        426,        1,   518400, 0x6fd028fa
     ++0,        426,        426,        1,   518400, 0x01f80e9d
       1,   85225000,   85225000,  2366000,     2585, 0x74d0048f
       0,        427,        427,        1,   518400, 0x01f80e9d
      -0,        438,        438,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        435,        435,        1,   518400, 0x01f80e9d
      +0,        436,        436,        1,   518400, 0x01f80e9d
      +0,        437,        437,        1,   518400, 0x01f80e9d
     -+0,        438,        438,        1,   518400, 0x01f80e9d
     ++0,        438,        438,        1,   518400, 0xb48d90c0
       1,   87652000,   87652000,  1831000,      634, 0x8832fda1
       0,        439,        439,        1,   518400, 0xb48d90c0
      -0,        447,        447,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        491,        491,        1,   518400, 0xb8a323e4
      +0,        492,        492,        1,   518400, 0xb8a323e4
      +0,        493,        493,        1,   518400, 0xb8a323e4
     -+0,        494,        494,        1,   518400, 0xb8a323e4
     ++0,        494,        494,        1,   518400, 0xc43518ba
       1,   98872000,   98872000,  2161000,     1875, 0x9002ef71
       0,        495,        495,        1,   518400, 0xc43518ba
      -0,        505,        505,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        847,        847,        1,   518400, 0x03ea0e90
      +0,        848,        848,        1,   518400, 0x03ea0e90
      +0,        849,        849,        1,   518400, 0x03ea0e90
     -+0,        850,        850,        1,   518400, 0x03ea0e90
     ++0,        850,        850,        1,   518400, 0x8780239e
       1,  170035000,  170035000,  1524000,     1279, 0xb6c2dafe
       0,        851,        851,        1,   518400, 0x8780239e
      -0,        858,        858,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        960,        960,        1,   518400, 0x59f4e72f
      +0,        961,        961,        1,   518400, 0x59f4e72f
      +0,        962,        962,        1,   518400, 0x59f4e72f
     -+0,        963,        963,        1,   518400, 0x59f4e72f
     ++0,        963,        963,        1,   518400, 0x9d5b9d69
       1,  192640000,  192640000,  1763000,     2506, 0xa458d6d4
       0,        964,        964,        1,   518400, 0x9d5b9d69
      -0,        972,        972,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,        993,        993,        1,   518400, 0x25113966
      +0,        994,        994,        1,   518400, 0x25113966
      +0,        995,        995,        1,   518400, 0x25113966
     -+0,        996,        996,        1,   518400, 0x25113966
     ++0,        996,        996,        1,   518400, 0x2dc83609
       1,  199230000,  199230000,  1627000,     1846, 0xeea8c599
       0,        997,        997,        1,   518400, 0x2dc83609
      -0,       1004,       1004,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video
      +0,       1136,       1136,        1,   518400, 0x81e977b2
      +0,       1137,       1137,        1,   518400, 0x81e977b2
      +0,       1138,       1138,        1,   518400, 0x81e977b2
     -+0,       1139,       1139,        1,   518400, 0x81e977b2
     ++0,       1139,       1139,        1,   518400, 0xb046dd30
       1,  227834000,  227834000,  1262000,     1264, 0xc1d9fc57
       0,       1140,       1140,        1,   518400, 0xb046dd30
      -0,       1145,       1145,        1,   518400, 0xbab197ea
     @@ tests/ref/fate/sub2video_basic
      -0,       8996,       8996,        1,  1382400, 0xda2ceb55
      -0,       9027,       9027,        1,  1382400, 0x00000000
      +#sar 0: 1/1
     -+0,          0,          0,        1,  1382400, 0x00000000
      +0,        662,        662,        1,  1382400, 0xc637b893
      +0,        663,        663,        1,  1382400, 0xc637b893
      +0,        664,        664,        1,  1382400, 0xc637b893
     @@ tests/ref/fate/sub2video_time_limited
      -#sar 0: 0/1
      -0,          2,          2,        1,  8294400, 0x00000000
      +#sar 0: 1/1
     -+0,          0,          0,        1,  8294400, 0x00000000
     -+0,          1,          1,        1,  8294400, 0x00000000
     ++0,          0,          0,        1,  8294400, 0xa87c518f
     ++0,          1,          1,        1,  8294400, 0xa87c518f
       0,          2,          2,        1,  8294400, 0xa87c518f
      +0,          3,          3,        1,  8294400, 0xa87c518f
      +0,          4,          4,        1,  8294400, 0xa87c518f
 22:  d3cdd2a0e2 <  -:  ---------- avutil/ass_split: Add parsing of hard-space tags (\h)
 23:  5fca566749 <  -:  ---------- avcodec/webvttenc: convert hard-space tags to &nbsp;
 24:  93b469b8d0 <  -:  ---------- doc/APIchanges: update for subtitle filtering changes
 25:  0c279550d6 <  -:  ---------- avcodec/webvttenc: Don't encode drawing codes and empty lines
 26:  03e1a98e08 ! 23:  a66debd96e avcodec/dvbsubdec: Fix conditions for fallback to default resolution
     @@ libavcodec/dvbsubdec.c
       
       #define cm (ff_crop_tab + MAX_NEG_CROP)
       
     -@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx,
     +@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx, AVSubtitle *sub,
           int segment_length;
           int i;
           int ret = 0;
     @@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx,
       
           ff_dlog(avctx, "DVB sub packet:\n");
       
     -@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx,
     +@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx, AVSubtitle *sub,
                   switch (segment_type) {
                   case DVBSUB_PAGE_SEGMENT:
                       ret = dvbsub_parse_page_segment(avctx, p, segment_length, sub, got_sub_ptr);
     @@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx,
                       break;
                   default:
                       ff_dlog(avctx, "Subtitling segment type 0x%x, page id %d, length %d\n",
     -@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx,
     +@@ libavcodec/dvbsubdec.c: static int dvbsub_decode(AVCodecContext *avctx, AVSubtitle *sub,
       
               p += segment_length;
           }