[FFmpeg-devel,v2,00/26] Subtitle Filtering 2022

Message ID	pull.18.v2.ffstaging.FFmpeg.1642646916.ffmpegagent@gmail.com
Headers	show Delivered-To: ffmpegpatchwork2@gmail.com Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Message-Id: <pull.18.v2.ffstaging.FFmpeg.1642646916.ffmpegagent@gmail.com> In-Reply-To: <pull.18.ffstaging.FFmpeg.1642122813.ffmpegagent@gmail.com> References: <pull.18.ffstaging.FFmpeg.1642122813.ffmpegagent@gmail.com> From: ffmpegagent <ffmpegagent@gmail.com> Date: Thu, 20 Jan 2022 02:48:10 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH v2 00/26] Subtitle Filtering 2022 Precedence: list Reply-To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> Cc: Michael Niedermayer <michael@niedermayer.cc>, softworkz <softworkz@hotmail.com>, Andreas Rheinhardt <andreas.rheinhardt@outlook.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" <ffmpeg-devel-bounces@ffmpeg.org>

Subtitle Filtering 2022 ======================= This is a substantial update to the earlier subtitle filtering patch series. A primary goal has been to address others' concerns as much as possible on one side and to provide more clarity and control over the way things are working. Clarity is is specifically important to allow for a better understanding of the need for a subtitle start pts value that can be different from the frame's pts value. This is done by refactoring the subtitle timing fields in AVFrame, adding a frame field to indicate repeated subtitle frames, and finally the full removal of the heartbeat functionality, replaced by a new 'subfeed' filter that provides different modes for arbitrating subtitle frames in a filter graph. Finally, each subtitle filter's documentation has been amended by a section describing the filter's timeline behavior (in v3 update). The update also includes major improvements to graphicsub2text and lots of other details. Versioning is restarting at v1 due to the new submission procedure. Changes in v2 ============= * added .gitattributes file to enforce binary diffs for the test refs that cannot be applied when being sent via e-mail * perform filter graph re-init due to subtitle "frame size" change only when the size was unknown before and not set via -canvas_size * overlaytextsubs: Make sure to request frames on the subtitle input * avfilter/splitcc: Start parsing cc data on key frames only * avcodec/webvttenc: Don't encode ass drawing codes and empty lines * stripstyles: fix mem leak * gs2t: improve color detection * gs2t: empty frames must not be skipped * subfeed: fix name * textmod: preserve margins * added .gitattributes file to enforce binary diffs for the test refs that cannot be applied when being sent via e-mail * perform filter graph re-init due to subtitle "frame size" change only when the size was unknown before and not set via -canvas_size * avcodec/dvbsubdec: Fix conditions for fallback to default resolution * Made changes suggested by Andreas * Fixed failing command line reported by Michael Changes from previous version v24: AVFrame ======= * Removed sub_start_time The start time is now added to the subtitle start_pts during decoding The sub_end_time field is adjusted accordingly * Renamed sub_end_time to duration which it is effectively after removing the start_time * Added a sub-struct 'subtitle_timing' to av frame Contains subtitle_pts renamed to 'subtitle_timing.start_pts' and 'subtitle_timing.duration' * Change both fields to (fixed) time_base AV_TIMEBASE * add repeat_sub field provides a clear indication whether a subtitle frame is an actual subtitle event or a repeated subtitle frame in a filter graph Heartbeat Removal ================= * completely removed the earlier heartbeat implementation * filtering arbitration is now implemented in a new filter: 'subfeed' * subfeed will be auto-inserted for compatiblity with sub2video command lines * the new behavior is not exactly identical to the earlier behavior, but it basically allows to achieve the same results New 'subfeed' Filter ==================== * a versatile filter for solving all kinds of problems with subtile frame flow in filter graphs * Can be inserted at any position in a graph * Auto-inserted for sub2video command lines (in repeat-mode) * Allows duration fixup delay input frames with unknown duration and infer duration from start of subsequent frame * Provides multiple modes of operation: * repeat mode (default) Queues input frames Outputs frames at a fixed (configurable) rate Either sends a matching input frame (repeatedly) or empty frames otherwise * scatter mode similar to repeat mode, but splits input frames by duration into small segments with same content * forward mode No fixed output rate Useful in combination with duration fixup or overlap fixup ffmpeg Tool Changes =================== * delay subtitle output stream initialization (like for audio and video) This is needed for example when a format header depends on having received an initial frame to derive certain header values from * decoding: set subtitle frame size from decoding context * re-init graph when subtitle size changes * always insert subscale filter for sub2video command lines (to ensure correct scaling) Subtitle Encoding ================= * ignore repeated frames for encoding based on repeat_sub field in AVFrame * support multi-area encoding for text subtitles Subtitle OCR can create multiple areas at different positions. Previously, the texts were always squashed into a single area ('subtitle rect'), which was not ideal. Multiple text areas are now generally supported: * ASS Encoder Changed to use the 'receive_packet' encoding API A single frame with multiple text areas will create multiple packets now * All other text subtitle encoders A newline is inserted between the text from multiple areas graphicsub2text (OCR) ===================== * enhanced preprocessing * using elbg algorithm for color quantization * detection and removal of text outlines * map-based identification of colors per word (text, outline, background) * add option for duration fixup * add option to dump preprocessing bitmaps * Recognize formatting and apply as ASS inline styles * per word(!) * paragraph alignment * positioning * font names * font size * font style (italic, underline, bold) * text color, outline color Other Filter Changes ==================== * all: Make sure to forward all link properties (time base, frame rate, w, h) where appropriate * overlaytextsubs: request frames on the subtitle input * overlaytextsubs: disable read-order checking * overlaytextsubs: improve implementation of render_latest_only * overlaytextsubs: ensure equal in/out video formats * splitcc: derive framerate from realtime_latency * graphicsub2video: implement caching of converted frames * graphicsub2video: use 1x1 output frame size as long as subtitle size is unknown (0x0) Plus a dozen of things I forgot.. softworkz (26): avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values avutil/frame: Prepare AVFrame for subtitle handling avcodec/subtitles: Introduce new frame-based subtitle decoding API avfilter/subtitles: Update vf_subtitles to use new decoding api avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing avcodec/subtitles: Replace deprecated enum values fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/avfilter: Fix hardcoded input index avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/splitcc: Add splitcc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles avfilter/subfeed: add subtitle feed filter avcodec/subtitles: Migrate subtitle encoders to frame-based API fftools/ffmpeg: Introduce subtitle filtering and new frame-based subtitle encoding avutil/ass_split: Add parsing of hard-space tags (\h) avcodec/webvttenc: convert hard-space tags to   doc/APIchanges: update for subtitle filtering changes avcodec/webvttenc: Don't encode drawing codes and empty lines avcodec/dvbsubdec: Fix conditions for fallback to default resolution configure | 7 +- doc/APIchanges | 24 + doc/filters.texi | 756 ++++++++++ fftools/ffmpeg.c | 501 +++---- fftools/ffmpeg.h | 13 +- fftools/ffmpeg_filter.c | 235 +++- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 +- fftools/ffprobe.c | 47 +- libavcodec/Makefile | 56 +- libavcodec/ass.h | 144 +- libavcodec/assdec.c | 4 +- libavcodec/assenc.c | 191 ++- libavcodec/avcodec.h | 32 +- libavcodec/ccaption_dec.c | 19 +- libavcodec/codec_desc.c | 11 + libavcodec/codec_desc.h | 8 + libavcodec/decode.c | 60 +- libavcodec/dvbsubdec.c | 53 +- libavcodec/dvbsubenc.c | 96 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 102 +- libavcodec/encode.c | 61 +- libavcodec/internal.h | 16 + libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 126 +- libavcodec/mpl2dec.c | 2 +- libavcodec/pgssubdec.c | 2 +- libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 116 +- libavcodec/subviewerdec.c | 2 +- libavcodec/tests/avcodec.c | 2 - libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 114 +- libavcodec/utils.c | 184 +++ libavcodec/version.h | 2 +- libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 127 +- libavcodec/xsubdec.c | 2 +- libavcodec/xsubenc.c | 88 +- libavfilter/Makefile | 16 + libavfilter/allfilters.c | 14 + libavfilter/avfilter.c | 30 +- libavfilter/avfilter.h | 11 + libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 54 + libavfilter/buffersink.h | 7 + libavfilter/buffersrc.c | 72 + libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 22 + libavfilter/formats.h | 3 + libavfilter/internal.h | 19 +- libavfilter/sf_graphicsub2text.c | 1132 +++++++++++++++ libavfilter/sf_splitcc.c | 395 ++++++ libavfilter/sf_stripstyles.c | 211 +++ libavfilter/sf_subfeed.c | 366 +++++ libavfilter/sf_subscale.c | 884 ++++++++++++ libavfilter/sf_textmod.c | 710 ++++++++++ libavfilter/subtitles.c | 63 + libavfilter/subtitles.h | 44 + libavfilter/vf_overlaygraphicsubs.c | 765 ++++++++++ libavfilter/vf_overlaytextsubs.c | 678 +++++++++ libavfilter/vf_subtitles.c | 56 +- libavutil/Makefile | 4 + {libavcodec => libavutil}/ass.c | 91 +- libavutil/ass_internal.h | 135 ++ {libavcodec => libavutil}/ass_split.c | 37 +- .../ass_split_internal.h | 32 +- libavutil/frame.c | 211 ++- libavutil/frame.h | 85 +- libavutil/subfmt.c | 45 + libavutil/subfmt.h | 115 ++ libavutil/version.h | 3 +- tests/ref/fate/.gitattributes | 3 + tests/ref/fate/filter-overlay-dvdsub-2397 | 182 +-- tests/ref/fate/mov-mp4-ttml-dfxp | 8 +- tests/ref/fate/mov-mp4-ttml-stpp | 8 +- tests/ref/fate/sub-dvb | 162 ++- tests/ref/fate/sub-textenc | 10 +- tests/ref/fate/sub-ttmlenc | 8 +- tests/ref/fate/sub-webvttenc | 10 +- tests/ref/fate/sub2video | 1091 ++++++++++++++- tests/ref/fate/sub2video_basic | 1239 +++++++++++++++-- tests/ref/fate/sub2video_time_limited | 78 +- 91 files changed, 11076 insertions(+), 1391 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_splitcc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subfeed.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlaygraphicsubs.c create mode 100644 libavfilter/vf_overlaytextsubs.c rename {libavcodec => libavutil}/ass.c (65%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (93%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h create mode 100644 tests/ref/fate/.gitattributes base-commit: c936c319bd54f097cc1d75b1ee1c407d53215d71 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-18%2Fsoftworkz%2Fsubmit_subfiltering-v2 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-18/softworkz/submit_subfiltering-v2 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/18 Range-diff vs v1: 1: 13b9a26b25 = 1: 13b9a26b25 avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values 2: 54ed1290d2 = 2: 54ed1290d2 avutil/frame: Prepare AVFrame for subtitle handling 3: c894aefe33 ! 3: 840683a6e1 avcodec/subtitles: Introduce new frame-based subtitle decoding API @@ Metadata ## Commit message ## avcodec/subtitles: Introduce new frame-based subtitle decoding API - - Add avcodec_decode_subtitle3 which takes subtitle frames, - serving as compatibility shim to legacy subtitle decoding + - Modify avcodec_send_packet() to support subtitles via the regular + frame based decoding API + - Add decode_subtitle_shim() which takes subtitle frames, + and serves as a compatibility shim to the legacy subtitle decoding + API until all subtitle decoders are migrated to the frame-based API - Add additional methods for conversion between old and new API Signed-off-by: softworkz <softworkz@hotmail.com> @@ libavcodec/decode.c: int attribute_align_arg avcodec_send_packet(AVCodecContext return AVERROR(EINVAL); + if (avctx->codec_type == AVMEDIA_TYPE_SUBTITLE) ++ // this does not exactly implement the avcodec_send_packet/avcodec_receive_frame API ++ // but we know that no subtitle decoder produces multiple AVSubtitles per packet through ++ // the legacy API, and this will be changed when migrating the subtitle decoders ++ // to the frame based decoding api + return decode_subtitle_shim(avctx, avci->buffer_frame, avpkt); + av_packet_unref(avci->buffer_pkt); @@ libavcodec/internal.h: int ff_int_from_list_or_default(void *ctx, const char * v + * the legacy subtitle API. + */ +int ff_frame_get_subtitle(AVSubtitle* sub, AVFrame* frame); -+ -+#if defined(_WIN32) && CONFIG_SHARED && !defined(BUILDING_avcodec) -+# define av_export_avcodec __declspec(dllimport) -+#else -+# define av_export_avcodec -+#endif + #endif /* AVCODEC_INTERNAL_H */ 4: 24d5ed19fe = 4: e326e96492 avfilter/subtitles: Update vf_subtitles to use new decoding api 5: baf6ca3dbc = 5: 20dc27a6c6 avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing 7: 34be31a7eb ! 6: 3de0be7980 avcodec/subtitles: Replace deprecated enum values @@ libavcodec/dvdsubdec.c: static int decode_dvd_subtitles(DVDSubContext *ctx, AVSu } ## libavcodec/dvdsubenc.c ## -@@ libavcodec/dvdsubenc.c: static int encode_dvd_subtitles(AVCodecContext* avctx, AVPacket* avpkt, +@@ libavcodec/dvdsubenc.c: static int encode_dvd_subtitles(AVCodecContext *avctx, + if (rects == 0 || !h->rects) return AVERROR(EINVAL); - for (i = 0; i < rects; i++) -- if (frame->subtitle_areas[i]->type != SUBTITLE_BITMAP) { -+ if (frame->subtitle_areas[i]->type != AV_SUBTITLE_FMT_BITMAP) { +- if (h->rects[i]->type != SUBTITLE_BITMAP) { ++ if (h->rects[i]->type != AV_SUBTITLE_FMT_BITMAP) { av_log(avctx, AV_LOG_ERROR, "Bitmap subtitle required\n"); return AVERROR(EINVAL); } 8: a13ddfc7b5 = 7: 05e3f2a6fe fftools/play,probe: Adjust for subtitle changes 9: b9501d32eb = 8: a86457f868 avfilter/subtitles: Add subtitles.c for subtitle frame allocation 10: f7df7f8d06 = 9: 9fec875c51 avfilter/avfilter: Handle subtitle frames 11: 76a2364048 = 10: 50bf6c36be avfilter/avfilter: Fix hardcoded input index 12: fdc41f4474 = 11: d1ef5004aa avfilter/sbuffer: Add sbuffersrc and sbuffersink filters 13: b47404812f = 12: f4af184a7b avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters 14: 453bef0c34 ! 13: 47b045ad74 avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters @@ libavfilter/vf_overlaytextsubs.c (new) + int detect_change = 0; + ASS_Image *image; + -+ ff_request_frame(ctx->inputs[1]); + -+ int64_t time_ms = (int64_t)((double)frame->pts * av_q2d(inlink->time_base) * 1000); ++ int64_t time_ms = (int64_t)((double)frame->pts * av_q2d(inlink->time_base) * 1000); ++ int64_t time_ms1 = (int64_t)((double)ctx->inputs[1]->current_pts * av_q2d(ctx->inputs[1]->time_base) * 1000); ++ ++ if (time_ms1 < time_ms + 1000) ++ ff_request_frame(ctx->inputs[1]); ++ ++ av_log(ctx, AV_LOG_DEBUG, "filter_video_frame - video: %"PRId64"ms sub: %"PRId64"ms rel %d\n", time_ms, time_ms1, (time_ms1 < time_ms)); + + ff_mutex_lock(&s->mutex); + image = ass_render_frame(s->renderer, s->track, time_ms, &detect_change); @@ libavfilter/vf_overlaytextsubs.c (new) + TextSubsContext *s = ctx->priv; + const int64_t start_time = av_rescale_q(frame->subtitle_timing.start_pts, AV_TIME_BASE_Q, av_make_q(1, 1000)); + const int64_t duration = av_rescale_q(frame->subtitle_timing.duration, AV_TIME_BASE_Q, av_make_q(1, 1000)); ++ const int64_t frame_time = (int64_t)((double)frame->pts * av_q2d(inlink->time_base) * 1000); + + // Postpone header processing until we receive a frame with content + if (!s->got_header && frame->num_subtitle_areas > 0) + process_header(ctx, frame); + ++ av_log(ctx, AV_LOG_DEBUG, "filter_subtitle_frame dur: %"PRId64"ms frame: %"PRId64"ms sub: %"PRId64"ms repeat_sub %d\n", duration, frame_time, start_time, frame->repeat_sub); ++ + if (frame->repeat_sub) + goto exit; + @@ libavfilter/vf_overlaytextsubs.c (new) + + if (s->track->events[i].Duration > diff) + s->track->events[i].Duration = diff; -+ + } + } + 15: efdaddc322 ! 14: 6f200be0c3 avfilter/textmod: Add textmod, censor and show_speaker filters @@ libavfilter/sf_textmod.c (new) + + av_bprint_finalize(&pbuf, &text); + -+ result = avpriv_ass_get_dialog(dialog->readorder, dialog->layer, dialog->style, dialog->name, text); ++ result = avpriv_ass_get_dialog_ex(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->margin_l, dialog->margin_r, dialog->margin_v, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); @@ libavfilter/sf_textmod.c (new) + if (!text) + return NULL; + -+ result = avpriv_ass_get_dialog(dialog->readorder, dialog->layer, dialog->style, dialog->name, text); ++ result = avpriv_ass_get_dialog_ex(dialog->readorder, dialog->layer, dialog->style, dialog->name, dialog->margin_l, dialog->margin_r, dialog->margin_v, text); + + av_free(text); + avpriv_ass_free_dialog(&dialog); 16: 54af3f8d07 ! 15: ab8736ac43 avfilter/stripstyles: Add stripstyles filter @@ libavfilter/sf_stripstyles.c (new) + if (!dialog) + return NULL; + -+ if (s->select_layer >= 0 && dialog->layer != s->select_layer) ++ if (s->select_layer >= 0 && dialog->layer != s->select_layer) { ++ avpriv_ass_free_dialog(&dialog); + return NULL; ++ } + + dlg_ctx.ss_ctx = s; + 17: e18f10394c ! 16: 5369aca080 avfilter/splitcc: Add splitcc filter for closed caption handling @@ libavfilter/sf_splitcc.c (new) + AVFrame *empty_sub_frame; + int new_frame; + int64_t next_repetition_pts; ++ int had_keyframe; + AVBufferRef *subtitle_header; + int use_cc_styles; + int real_time; @@ libavfilter/sf_splitcc.c (new) + AVFilterLink *inlink0 = ctx->inputs[0]; + AVFilterLink *outlink0 = ctx->outputs[0]; + AVFilterLink *outlink1 = ctx->outputs[1]; -+ static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NB }; ++ static const enum AVSubtitleType subtitle_fmts[] = { AV_SUBTITLE_FMT_ASS, AV_SUBTITLE_FMT_NONE }; + int ret; + + /* set input0 video formats */ @@ libavfilter/sf_splitcc.c (new) + + sd = av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC); + -+ if (sd) { ++ if (sd && (s->had_keyframe || frame->key_frame)) { + int got_output = 0; + ++ s->had_keyframe = 1; + pkt = av_packet_alloc(); + pkt->buf = av_buffer_ref(sd->buf); + if (!pkt->buf) { @@ libavfilter/sf_splitcc.c (new) + goto fail; + } + -+ pkt->data = sd->data; -+ pkt->size = (int)sd->size; -+ pkt->pts = frame->pts; ++ pkt->data = pkt->buf->data; ++ pkt->size = pkt->buf->size; ++ pkt->pts = av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q); + + sub_out = ff_get_subtitles_buffer(outlink1, AV_SUBTITLE_FMT_ASS); + if (!sub_out) { @@ libavfilter/sf_splitcc.c (new) + + ret = decode(s->cc_dec, sub_out, &got_output, pkt); + -+ if (ret < 0) ++ ////if (got_output) { ++ //// av_log(inlink->dst, AV_LOG_INFO, "CC Packet PTS: %"PRId64" got_output: %d out_frame_pts: %"PRId64" out_sub_pts: %"PRId64"\n", pkt->pts, got_output, sub_out->pts, sub_out->subtitle_pts); ++ ////} ++ ++ av_packet_free(&pkt); ++ ++ if (ret < 0) { ++ av_log(inlink->dst, AV_LOG_ERROR, "Decode error: %d \n", ret); + goto fail; ++ } + + if (got_output) { + sub_out->pts = frame->pts; 18: 2cf4d42167 ! 17: 43a20d1024 avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) @@ doc/filters.texi: ffmpeg -i "https://streams.videolan.org/ffmpeg/mkv_subtitles.m + +Note: For this to work, you need to have the data file 'eng.traineddata' in a 'tessdata' subfolder (see above). +@example -+ffmpeg ffmpeg -loglevel verbose -i "https://streams.videolan.org/streams/ts/video_subs_ttxt%2Bdvbsub.ts" -filter_complex "[0:13]graphicsub2text=ocr_mode=both" -c:s ass -y output.mkv ++ffmpeg -loglevel verbose -i "https://streams.videolan.org/streams/ts/video_subs_ttxt%2Bdvbsub.ts" -filter_complex "[0:13]graphicsub2text=delay_when_no_duration=1" -c:s ass -y output.mkv +@end example +@end itemize + @@ libavfilter/sf_graphicsub2text.c (new) + for (int y = top; y < bottom; y += 3) { + uint8_t *p = area->buf[0]->data + (y * area->linesize[0]) + left; + uint8_t *porig = original_area->buf[0]->data + (y * original_area->linesize[0]) + left; ++ uint8_t current_index = 255; + + for (int x = left; x < right; x++, p++, porig++) { -+ if (*p == bg_color_index) -+ bg_score[*porig]++; -+ if (*p == text_color_index) -+ text_score[*porig]++; -+ if (*p == outline_color_index) -+ outline_score[*porig]++; ++ ++ if (*p == current_index) { ++ if (*p == bg_color_index) ++ bg_score[*porig]++; ++ if (*p == text_color_index) ++ text_score[*porig]++; ++ if (*p == outline_color_index) ++ outline_score[*porig]++; ++ } ++ ++ current_index = *p; + } + } + @@ libavfilter/sf_graphicsub2text.c (new) + + frame_sent = 1; + s->last_subtitle_pts = frame->subtitle_timing.start_pts; -+ -+ if (frame->num_subtitle_areas == 0) { -+ // No need to forward this empty frame -+ av_frame_free(&frame); -+ return 0; -+ } + } + -+ if (frame->repeat_sub && frame->subtitle_timing.start_pts == s->last_subtitle_pts) { ++ if (frame->repeat_sub) { + // Ignore repeated frame + av_frame_free(&frame); + return 0; @@ libavfilter/sf_graphicsub2text.c (new) + + // When decoders can't determine the end time, they are setting it either to UINT32_NAX + // or 30s (dvbsub). -+ if (s->delay_when_no_duration && frame->num_subtitle_areas > 0 && frame->subtitle_timing.duration >= ms_to_avtb(29000)) { ++ if (s->delay_when_no_duration && frame->subtitle_timing.duration >= ms_to_avtb(29000)) { + // Can't send it without end time, wait for the next frame to determine the end_display time + s->pending_frame = frame; + 19: c51d6a77da = 18: 0d3c46a68f avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles 20: ad07dbf7c2 ! 19: 2c20886389 avfilter/subfeed: add subtitle feed filter @@ libavfilter/sf_subfeed.c (new) + +const AVFilter ff_sf_subfeed = { + .name = "subfeed", -+ .description = NULL_IF_CONFIG_SMALL("Scatter subtitle events by duration"), ++ .description = NULL_IF_CONFIG_SMALL("Control subtitle frame timing and flow in a filtergraph"), + .init = init, + .uninit = uninit, + .priv_size = sizeof(SubFeedContext), 6: 808f589488 ! 20: 64ce976c19 avcodec/subtitles: Migrate subtitle encoders to frame-based API @@ libavcodec/dvdsubenc.c: static void copy_rectangle(AVSubtitleRect *dst, AVSubtit return AVERROR(EINVAL); + for (i = 0; i < rects; i++) -- if (h->rects[i]->type != SUBTITLE_BITMAP) { -+ if (frame->subtitle_areas[i]->type != SUBTITLE_BITMAP) { +- if (h->rects[i]->type != AV_SUBTITLE_FMT_BITMAP) { ++ if (frame->subtitle_areas[i]->type != AV_SUBTITLE_FMT_BITMAP) { av_log(avctx, AV_LOG_ERROR, "Bitmap subtitle required\n"); return AVERROR(EINVAL); } @@ libavcodec/encode.c: fail: return ret; } -+/** -+ * \brief -+ * \param avctx -+ * \param buf q -+ * \param buf_size -+ * \param sub -+ * \return -+ */ - int avcodec_encode_subtitle(AVCodecContext *avctx, uint8_t *buf, int buf_size, - const AVSubtitle *sub) +-int avcodec_encode_subtitle(AVCodecContext *avctx, uint8_t *buf, int buf_size, +- const AVSubtitle *sub) ++int avcodec_encode_subtitle(AVCodecContext *avctx, uint8_t *buf, int buf_size, const AVSubtitle *sub) { - int ret; + int ret = 0, got_packet = 0; @@ libavcodec/encode.c: fail: + if (ret < 0) + goto exit; + -+ ret = avctx->codec->encode2(avctx, avpkt, frame, &got_packet); ++ ret = avcodec_send_frame(avctx, frame); ++ if (ret < 0) ++ goto exit; ++ ++ ret = avcodec_receive_packet(avctx, avpkt); ++ ++ if (ret < 0 && ret != AVERROR(EAGAIN)) ++ goto exit; ++ ++ //ret = avctx->codec->encode2(avctx, avpkt, frame, &got_packet); + avctx->frame_number++; + -+ if (got_packet) { ++ if (avpkt->size) { + if (avpkt->size > buf_size) { + ret = AVERROR_BUFFER_TOO_SMALL; + goto exit; 21: 00643582cf ! 21: 56a162b3a4 fftools/ffmpeg: Introduce subtitle filtering new frame-based subtitle encoding @@ Metadata Author: softworkz <softworkz@hotmail.com> ## Commit message ## - fftools/ffmpeg: Introduce subtitle filtering new frame-based subtitle encoding + fftools/ffmpeg: Introduce subtitle filtering and new frame-based subtitle encoding This commit actually enables subtitle filtering in ffmpeg by sending and receiving subtitle frames to and from a filtergraph. @@ fftools/ffmpeg.c: static int ifilter_has_all_input_formats(FilterGraph *fg) } return 1; @@ fftools/ffmpeg.c: static int ifilter_send_frame(InputFilter *ifilter, AVFrame *frame, int keep_ref - ifilter->channel_layout != frame->channel_layout; - break; case AVMEDIA_TYPE_VIDEO: -+ case AVMEDIA_TYPE_SUBTITLE: need_reinit |= ifilter->width != frame->width || ifilter->height != frame->height; + + if (need_reinit) ++ need_reinit = 1; ++ break; ++ case AVMEDIA_TYPE_SUBTITLE: ++ need_reinit |= ifilter->width != frame->width || ++ ifilter->height != frame->height; ++ ++ need_reinit &= (ifilter->width == 0 || ifilter->height == 0); ++ ++ if (need_reinit) + need_reinit = 1; break; } @@ fftools/ffmpeg.c: fail: + AVFrame *decoded_frame; + AVCodecContext *avctx = ist->dec_ctx; + int i = 0, ret = 0, err = 0; - -- check_decode_result(NULL, got_output, ret); ++ + if (!ist->decoded_frame && !(ist->decoded_frame = av_frame_alloc())) + return AVERROR(ENOMEM); + decoded_frame = ist->decoded_frame; + decoded_frame->type = AVMEDIA_TYPE_SUBTITLE; + decoded_frame->format = avcodec_descriptor_get_subtitle_format(avctx->codec_descriptor); -+ + +- check_decode_result(NULL, got_output, ret); + if (!ist->subtitle_header && avctx->subtitle_header && avctx->subtitle_header_size > 0) { + ist->subtitle_header = av_buffer_allocz(avctx->subtitle_header_size + 1); + if (!ist->subtitle_header) 22: b48816fb4c ! 22: 1a0c6e01f3 avutil/ass_split: Add parsing of hard-space tags (\h) @@ libavutil/ass_split.c: int avpriv_ass_split_override_codes(const ASSCodesCallbac buf++; while (*buf == '\\') { + ## tests/ref/fate/.gitattributes (new) ## +@@ ++sub-textenc -diff ++sub-ttmlenc -diff ++sub-webvttenc -diff + ## tests/ref/fate/mov-mp4-ttml-dfxp ## @@ -2e7e01c821c111466e7a2844826b7f6d *tests/data/fate/mov-mp4-ttml-dfxp.mp4 23: 10c1404272 = 23: 44b4e203d8 avcodec/webvttenc: convert hard-space tags to   24: c0011334ce = 24: 5773f2a1ff doc/APIchanges: update for subtitle filtering changes -: ---------- > 25: 7ab6dedf80 avcodec/webvttenc: Don't encode drawing codes and empty lines -: ---------- > 26: 217d96c39d avcodec/dvbsubdec: Fix conditions for fallback to default resolution

[FFmpeg-devel,v2,00/26] Subtitle Filtering 2022

Message