From patchwork Fri Jan 14 01:13:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 33562 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:cd86:0:0:0:0:0 with SMTP id d128csp984711iog; Thu, 13 Jan 2022 17:13:49 -0800 (PST) X-Google-Smtp-Source: ABdhPJxRFaQ0c78OoBlS0W3zDpRNcA2QXHKP6teLd+ByyIFkDj/LiC5XVFD5GAKXs+UGTI84M+np X-Received: by 2002:a17:906:1dc3:: with SMTP id v3mr372241ejh.264.1642122829078; Thu, 13 Jan 2022 17:13:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1642122829; cv=none; d=google.com; s=arc-20160816; b=IPJlOduFSiX1TrdJYugGBUo1+I7RyL6eBePciGqQ8ttxYZYNdZ59D0454i/Xwjm/ij dRr0B+Crj9oWMrP6e79mVNJUm04VNP2SSTGTgZnQJGunMnYo2Lbdg7lZv7HFqPhlazRD 2jzzp163Z7+X/PG8VJOfhagBM3Z5KR78Njo0vFgl9vTql3JtK7q1y1GkEOhb8ZC1xL9L YSIV5GoOOuwJHy6f8FN/jgcVh4il5qtpwOJNyp7ll/ZfK2U6Q+2eOjrQ9LT8NRzOTEZt K6ZZhIoLS88YwHZ2bKI+oBZ2wiCm9Qmp6Pl3SdiKkzOoBzxsgkXqI7Q/OnBZAEyhu2jy mqNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:from:message-id :dkim-signature:delivered-to; bh=xPX0hFwpcO9q3kxesuBrP9zu4JN4Me+ShEL67z1hhbU=; b=nIxkMxQVQLUOZYYyRHFL4hBdtToWvBP2viV+Hh6IekFVO3V9VAvQo3lXcm5H+tDG2b 87gdvzKQS9Quo+JNKRUxOtEipNcxbOFyhULjbLQNNhVkPeavuJkKXjf1m50gewoxs9MD DugRS8Bbt9gBSlcpBHxnljfy9/SUn8pAH535lKclOPX9dc3MCSEEzDYjXhjszB3VVE2R FhjSNM2MBc4yu/88VTSoiB3jQ7ZVBf5Ve9Dn3ZtTH/rJNKcYyM2UoSNf8v/Z0XalC4k/ /PYv79fyvDbJWOscpFGrFbncDLgQNCiE10Ac+Nw+KjEMdyhEoX+ikbI8OdaJATbvbGwa ViyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=D01MADKa; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id z7si2528601edb.35.2022.01.13.17.13.48; Thu, 13 Jan 2022 17:13:49 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=D01MADKa; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DD9FA68B3BD; Fri, 14 Jan 2022 03:13:43 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A2E3468B3BD for ; Fri, 14 Jan 2022 03:13:37 +0200 (EET) Received: by mail-pl1-f171.google.com with SMTP id t18so11913178plg.9 for ; Thu, 13 Jan 2022 17:13:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:from:date:subject:fcc:content-transfer-encoding :mime-version:to:cc; bh=fH8qgKvUWb4JZLEBjBYaa3ZAGXwBwjmFf9sBaHqxYiA=; b=D01MADKaXwhL1jQHyHt+p+TpIZh//4UgkKs12LLuj3jfSLFWuSN8y3lGN2tWRjrICr HLPFuTTbsff16oHMbTKWl0xE5mO5SWAcayJgEakP7TWxfTJSZeivga766v7WP+nwgTAw 8NyT6Kp8aBtIuX35HQTfG++720yva0e4JngLPZ4DeBGREz4gDsimPzRNQnwNE+HWp335 zOjMipG37vG9TmZuffy3tKflOIG7vhgb/jffgunv0jirvQE7yaB3OTPi38PBhdt9bZre zxdtPZajVk5qUR8G60F+47z9F4wMGRbL+qZaPMrO1SX4ZfjS5/96UhckNWB9DLi2Vvmt MjVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fH8qgKvUWb4JZLEBjBYaa3ZAGXwBwjmFf9sBaHqxYiA=; b=TvtTEPwNxRqtgkC+7exiAQpyp/XXFFdA4XkmhixOAPePQeq+VA6ySlug8R68X4gviO uK+Tn5hWVCFERAIf7Z9vpQGy7tz0UVEJ7mwVZSxk7GuxbSTClY0pX9P2ZID5isl/LZrU vR6W7M/Kw6msZ14qLZ8bCSZhEArqsWNu4G1jFGnbmxNS5izMr6yac3TCEEm7txkZQLsE 9oHOkL5KxhfpL1y77VaCzAXBAlVu2SLsWK1uPCkLSiNyIBmaroZZGgx/LYdf/RGR/4L4 N1U7K6VWJmG2V4sfz5FC4gWsrYYIAr4ydsF+2yc74UQM6+aF32+QZ+Y5MnStTNh6lZWx Ltnw== X-Gm-Message-State: AOAM533fS29tHPZ/4a4HZkDpkbpT7yvPIwzzwPsw4qzwErHl/JUBAEhR vCqVvoWRxKpFtmMRiA72mW6LMTiIiHePkg== X-Received: by 2002:a17:902:d2c8:b0:14a:55fb:cfe5 with SMTP id n8-20020a170902d2c800b0014a55fbcfe5mr7137597plc.51.1642122815897; Thu, 13 Jan 2022 17:13:35 -0800 (PST) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id z14sm3950246pfh.73.2022.01.13.17.13.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jan 2022 17:13:35 -0800 (PST) Message-Id: From: ffmpegagent Date: Fri, 14 Jan 2022 01:13:09 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH 00/24] Subtitle Filtering 2022 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: softworkz Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: rqwMGNlyDVYl Subtitle Filtering 2022 ======================= This is a substantial update to the earlier subtitle filtering patch series. A primary goal has been to address others' concerns as much as possible on one side and to provide more clarity and control over the way things are working. Clarity is is specifically important to allow for a better understanding of the need for a subtitle start pts value that can be different from the frame's pts value. This is done by refactoring the subtitle timing fields in AVFrame, adding a frame field to indicate repeated subtitle frames, and finally the full removal of the heartbeat functionality, replaced by a new 'subfeed' filter that provides different modes for arbitrating subtitle frames in a filter graph. Finally, each subtitle filter's documentation has been amended by a section describing the filter's timeline behavior (in v2 update). The update also includes major improvements to graphicsub2text and lots of other details. Versioning is restarting at v1 due to the new submission procedure. All changes from previous version v24: AVFrame ======= * Removed sub_start_time The start time is now added to the subtitle start_pts during decoding The sub_end_time field is adjusted accordingly * Renamed sub_end_time to duration which it is effectively after removing the start_time * Added a sub-struct 'subtitle_timing' to av frame Contains subtitle_pts renamed to 'subtitle_timing.start_pts' and 'subtitle_timing.duration' * Change both fields to (fixed) time_base AV_TIMEBASE * add repeat_sub field provides a clear indication whether a subtitle frame is an actual subtitle event or a repeated subtitle frame in a filter graph Heartbeat Removal ================= * completely removed the earlier heartbeat implementation * filtering arbitration is now implemented in a new filter: 'subfeed' * subfeed will be auto-inserted for compatiblity with sub2video command lines * the new behavior is not exactly identical to the earlier behavior, but it basically allows to achieve the same results New 'subfeed' Filter ==================== * a versatile filter for solving all kinds of problems with subtile frame flow in filter graphs * Can be inserted at any position in a graph * Auto-inserted for sub2video command lines (in repeat-mode) * Allows duration fixup delay input frames with unknown duration and infer duration from start of subsequent frame * Provides multiple modes of operation: * repeat mode (default) Queues input frames Outputs frames at a fixed (configurable) rate Either sends a matching input frame (repeatedly) or empty frames otherwise * scatter mode similar to repeat mode, but splits input frames by duration into small segments with same content * forward mode No fixed output rate Useful in combination with duration fixup or overlap fixup ffmpeg Tool Changes =================== * delay subtitle output stream initialization (like for audio and video) This is needed for example when a format header depends on having received an initial frame to derive certain header values from * decoding: set subtitle frame size from decoding context * re-init graph when subtitle size changes * always insert subscale filter for sub2video command lines (to ensure correct scaling) Subtitle Encoding ================= * ignore repeated frames for encoding based on repeat_sub field in AVFrame * support multi-area encoding for text subtitles Subtitle OCR can create multiple areas at different positions. Previously, the texts were always squashed into a single area ('subtitle rect'), which was not ideal. Multiple text areas are now generally supported: * ASS Encoder Changed to use the 'receive_packet' encoding API A single frame with multiple text areas will create multiple packets now * All other text subtitle encoders A newline is inserted between the text from multiple areas graphicsub2text (OCR) ===================== * enhanced preprocessing * using elbg algorithm for color quantization * detection and removal of text outlines * map-based identification of colors per word (text, outline, background) * add option for duration fixup * add option to dump preprocessing bitmaps * Recognize formatting and apply as ASS inline styles * per word(!) * paragraph alignment * positioning * font names * font size * font style (italic, underline, bold) * text color, outline color Other Filter Changes ==================== * all: Make sure to forward all link properties (time base, frame rate, w, h) where appropriate * overlaytextsubs: request frames on the subtitle input * overlaytextsubs: disable read-order checking * overlaytextsubs: improve implementation of render_latest_only * overlaytextsubs: ensure equal in/out video formats * splitcc: derive framerate from realtime_latency * graphicsub2video: implement caching of converted frames * graphicsub2video: use 1x1 output frame size as long as subtitle size is unknown (0x0) Plus a dozen of things I forgot.. softworkz (24): avcodec,avutil: Move enum AVSubtitleType to avutil, add new and deprecate old values avutil/frame: Prepare AVFrame for subtitle handling avcodec/subtitles: Introduce new frame-based subtitle decoding API avfilter/subtitles: Update vf_subtitles to use new decoding api avcodec,avutil: Move ass helper functions to avutil as avpriv_ and extend ass dialog parsing avcodec/subtitles: Migrate subtitle encoders to frame-based API avcodec/subtitles: Replace deprecated enum values fftools/play,probe: Adjust for subtitle changes avfilter/subtitles: Add subtitles.c for subtitle frame allocation avfilter/avfilter: Handle subtitle frames avfilter/avfilter: Fix hardcoded input index avfilter/sbuffer: Add sbuffersrc and sbuffersink filters avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2video filters avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters avfilter/textmod: Add textmod, censor and show_speaker filters avfilter/stripstyles: Add stripstyles filter avfilter/splitcc: Add splitcc filter for closed caption handling avfilter/graphicsub2text: Add new graphicsub2text filter (OCR) avfilter/subscale: Add filter for scaling and/or re-arranging graphical subtitles avfilter/subfeed: add subtitle feed filter fftools/ffmpeg: Introduce subtitle filtering new frame-based subtitle encoding avutil/ass_split: Add parsing of hard-space tags (\h) avcodec/webvttenc: convert hard-space tags to   doc/APIchanges: update for subtitle filtering changes configure | 7 +- doc/APIchanges | 24 + doc/filters.texi | 756 ++++++++++ fftools/ffmpeg.c | 493 +++---- fftools/ffmpeg.h | 13 +- fftools/ffmpeg_filter.c | 235 +++- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- fftools/ffplay.c | 102 +- fftools/ffprobe.c | 47 +- libavcodec/Makefile | 56 +- libavcodec/ass.h | 144 +- libavcodec/assdec.c | 4 +- libavcodec/assenc.c | 191 ++- libavcodec/avcodec.h | 32 +- libavcodec/ccaption_dec.c | 19 +- libavcodec/codec_desc.c | 11 + libavcodec/codec_desc.h | 8 + libavcodec/decode.c | 56 +- libavcodec/dvbsubdec.c | 2 +- libavcodec/dvbsubenc.c | 96 +- libavcodec/dvdsubdec.c | 2 +- libavcodec/dvdsubenc.c | 102 +- libavcodec/encode.c | 57 +- libavcodec/internal.h | 22 + libavcodec/jacosubdec.c | 2 +- libavcodec/libaribb24.c | 2 +- libavcodec/libzvbi-teletextdec.c | 14 +- libavcodec/microdvddec.c | 7 +- libavcodec/movtextdec.c | 3 +- libavcodec/movtextenc.c | 126 +- libavcodec/mpl2dec.c | 2 +- libavcodec/pgssubdec.c | 2 +- libavcodec/realtextdec.c | 2 +- libavcodec/samidec.c | 2 +- libavcodec/srtdec.c | 2 +- libavcodec/srtenc.c | 116 +- libavcodec/subviewerdec.c | 2 +- libavcodec/tests/avcodec.c | 2 - libavcodec/textdec.c | 4 +- libavcodec/ttmlenc.c | 114 +- libavcodec/utils.c | 184 +++ libavcodec/version.h | 2 +- libavcodec/webvttdec.c | 2 +- libavcodec/webvttenc.c | 100 +- libavcodec/xsubdec.c | 2 +- libavcodec/xsubenc.c | 88 +- libavfilter/Makefile | 16 + libavfilter/allfilters.c | 14 + libavfilter/avfilter.c | 30 +- libavfilter/avfilter.h | 11 + libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 54 + libavfilter/buffersink.h | 7 + libavfilter/buffersrc.c | 72 + libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 22 + libavfilter/formats.h | 3 + libavfilter/internal.h | 19 +- libavfilter/sf_graphicsub2text.c | 1132 +++++++++++++++ libavfilter/sf_splitcc.c | 385 +++++ libavfilter/sf_stripstyles.c | 209 +++ libavfilter/sf_subfeed.c | 366 +++++ libavfilter/sf_subscale.c | 884 ++++++++++++ libavfilter/sf_textmod.c | 710 ++++++++++ libavfilter/subtitles.c | 63 + libavfilter/subtitles.h | 44 + libavfilter/vf_overlaygraphicsubs.c | 765 ++++++++++ libavfilter/vf_overlaytextsubs.c | 671 +++++++++ libavfilter/vf_subtitles.c | 56 +- libavutil/Makefile | 4 + {libavcodec => libavutil}/ass.c | 91 +- libavutil/ass_internal.h | 135 ++ {libavcodec => libavutil}/ass_split.c | 37 +- .../ass_split_internal.h | 32 +- libavutil/frame.c | 211 ++- libavutil/frame.h | 85 +- libavutil/subfmt.c | 45 + libavutil/subfmt.h | 115 ++ libavutil/version.h | 3 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 182 +-- tests/ref/fate/mov-mp4-ttml-dfxp | 8 +- tests/ref/fate/mov-mp4-ttml-stpp | 8 +- tests/ref/fate/sub-dvb | 162 ++- tests/ref/fate/sub-textenc | 10 +- tests/ref/fate/sub-ttmlenc | 8 +- tests/ref/fate/sub-webvttenc | 10 +- tests/ref/fate/sub2video | 1091 ++++++++++++++- tests/ref/fate/sub2video_basic | 1239 +++++++++++++++-- tests/ref/fate/sub2video_time_limited | 78 +- 90 files changed, 10991 insertions(+), 1366 deletions(-) create mode 100644 libavfilter/sf_graphicsub2text.c create mode 100644 libavfilter/sf_splitcc.c create mode 100644 libavfilter/sf_stripstyles.c create mode 100644 libavfilter/sf_subfeed.c create mode 100644 libavfilter/sf_subscale.c create mode 100644 libavfilter/sf_textmod.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/vf_overlaygraphicsubs.c create mode 100644 libavfilter/vf_overlaytextsubs.c rename {libavcodec => libavutil}/ass.c (65%) create mode 100644 libavutil/ass_internal.h rename {libavcodec => libavutil}/ass_split.c (93%) rename libavcodec/ass_split.h => libavutil/ass_split_internal.h (86%) create mode 100644 libavutil/subfmt.c create mode 100644 libavutil/subfmt.h base-commit: c936c319bd54f097cc1d75b1ee1c407d53215d71 Published-As: https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-18%2Fsoftworkz%2Fsubmit_subfiltering-v1 Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-18/softworkz/submit_subfiltering-v1 Pull-Request: https://github.com/ffstaging/FFmpeg/pull/18