From patchwork Thu May 4 22:02:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 41476 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:dca6:b0:f3:34fa:f187 with SMTP id ky38csp13009pzb; Thu, 4 May 2023 14:08:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4AmKMyLEoukpp4vA/l2qxZk3klP0Mz2Mr3iU0EILd3tHVeOtNISDjfgKsSScYySJiggVkf X-Received: by 2002:a17:906:4fd1:b0:889:5686:486a with SMTP id i17-20020a1709064fd100b008895686486amr195413ejw.30.1683234483908; Thu, 04 May 2023 14:08:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683234483; cv=none; d=google.com; s=arc-20160816; b=gPupBFKA+DJjMD0NBz76tUkBx9eB4Jm+icws3oAXVJWdoboaHlhCBLcerDxFWDXJc3 a/PP18eZwbG+c3yJQesZ8kouC0UqFJt+fg17RF9ci+7Dy45Pj3nTlZWe6ANl+1ffSDks E6X2JBF2EOwJZHubcAQRi6j8RC40a14ZGFHi/4IHMbtkD000mXfgIjhkDM8pZS5T55Om s60qB6WGT0r+ojVOUcVVgTLK10xqrz0gKq8e90TQ1rS3WvTy7s6FGRJwO4x9oNCJnRNN 8vZrjsLX7yIdUyKM7J1tdUCzlsq/rb6fVNU/7GHfQnO0bGafv4ovTd2Ot88MQPTmO4Xo crHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=RhfARq+TsdbvsHUpS9ShGbmqkLo8aYDj8RqzOhYYibg=; b=Z8NODByxfycCLO/KI2tMwA4QDNsT3yXJuAruACKPoCuaButVc8fQgs6p7unLZRyM3w q53mLT7f8Uoqfe5OBmUwIbX6K4CDP0D1cWB7slEchVdQFVXfCoBScMiWBfhPPgujWa0+ Yz3L2IKkZZ2ao9EXLNH9RJNJQe9pGb0iztYKg6hAtDJ8F4YJyPY8no7fFl+He37cYSf8 XS57X21USBDrRk0jyC+Gi8cWxjCiYMBDuhgXrc91G3cLzys8uCIzisA5hZsmDrv1ZI+F r0XiwN1d2DM4ANaXoA/xx6bAzgCLIudqv4DCg/4XRyP6tvNzbhDPxDudE7mIapaVHb8I Zaug== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b="OZ3+/Gqp"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ec15-20020a170906b6cf00b009604f4a3fe4si54951ejb.53.2023.05.04.14.08.03; Thu, 04 May 2023 14:08:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b="OZ3+/Gqp"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 193D668C13E; Fri, 5 May 2023 00:06:51 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id B28D868C10D for ; Fri, 5 May 2023 00:06:41 +0300 (EEST) Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-74fc1452fbdso79025385a.2 for ; Thu, 04 May 2023 14:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1683234400; x=1685826400; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=KWa1d9LQ33UJDbR44HgPCk5kbWfV1BdRusHOBdIv6go=; b=OZ3+/Gqpjj7B6RIPS4Dag8OXarEgBeEmj+B3SFzma1ggJ4Hd2k+MYXnDyZ1bXTF0wg wav4Bno0sQ0ziOD6Y6XcIz81RetmnPmGC9HZHXsZBskBaZacmvt5u2QY1OeKZ2V9ZBGa XV9Q8H+/WADRfr02PCfNOygFaAOeT23hJLJR/heCpwo7OLBH5T9GgE4Gd9sA1abk1Z9Z ZavPCcNCh71XspDsSoYrmMyoFtIwNhPPCfzRuX1UQ7BHMW8nM7BvDcN5QnpaAL2aGySw 0vSOf1YeTBKkEPfArsBS+z/HcwmS0Q+0IPadktauwgAtivEeCtBlMKH2+euDjf9rdK+a 6T/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683234400; x=1685826400; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KWa1d9LQ33UJDbR44HgPCk5kbWfV1BdRusHOBdIv6go=; b=aw6Z746iiT+15HtzFBRLb7zGK5lTID4hw0M3w3hf+Wfe34vPi/9wZQoy4sEpv01qFc Dksa1mpM27fJkv2wP/0SD8Iqa4B8Jnm2C5bp/zQ5J3zCBEM+2ODv8cxCIiWKDusXeBXR IA+WVmrWyQwbJ2Z4PGH9NJDKlOBnqGQl/xN2gtzkffnJJdtdLJTCAOO1+mENEG5l/Bv1 haSq6vasJLNZ7J6PFbN4jPXxMSebvfDsmExx84juE+ff3v59rrQ4QbDwD47n0WazCpeg yD5ieIjmoSTLA22EoYoe4Wo9fTIx6WBeOOTNehMRjap5FCMKoNQykyhBPG004qmoJNAl tNNw== X-Gm-Message-State: AC+VfDwlwUmuEWZfOwRHUFPZ7YqCe46fodnrplgX4RCpmNji/xdvPYdq 7/psTKJw8HdQV0dBnrPFHIKM+oQSvoQOBdRwotc= X-Received: by 2002:a05:6214:e66:b0:616:516e:f3ea with SMTP id jz6-20020a0562140e6600b00616516ef3eamr18116643qvb.51.1683234400295; Thu, 04 May 2023 14:06:40 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id 4-20020a05620a070400b0074e21c3bc8asm67996qkc.126.2023.05.04.14.06.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 May 2023 14:06:39 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Thu, 4 May 2023 18:02:20 -0400 Message-Id: <1683237740-32743-7-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1683237740-32743-1-git-send-email-dheitmueller@ltnglobal.com> References: <1683237740-32743-1-git-send-email-dheitmueller@ltnglobal.com> Subject: [FFmpeg-devel] [PATCH v5 6/6] decklink_enc: add support for playout of 608 captions in MOV files X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: rVhiNTmjl32R Unlike other cases where the closed captions are embedded in the video stream as MPEG-2 userdata or H.264 SEI data, with MOV files the captions are often found on a separate "e608" subtitle track. Add support for playout of such files, leveraging the new ccfifo mechanism to ensure that they are embedded into VANC at the correct rate (since e608 packets often contain batches of multiple 608 pairs). Note this patch includes a new file named libavdevice/ccfifo.c, which allows the ccfifo functionality in libavfilter to be reused even if doing shared builds. This is the same approach used for log2_tab.c. Signed-off-by: Devin Heitmueller --- libavdevice/Makefile | 1 + libavdevice/ccfifo.c | 24 ++++++++++++++++ libavdevice/decklink_common.h | 3 ++ libavdevice/decklink_enc.cpp | 65 +++++++++++++++++++++++++++++++++++++++++++ libavdevice/decklink_enc_c.c | 2 +- 5 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 libavdevice/ccfifo.c diff --git a/libavdevice/Makefile b/libavdevice/Makefile index 8a62822..c304492 100644 --- a/libavdevice/Makefile +++ b/libavdevice/Makefile @@ -57,6 +57,7 @@ OBJS-$(CONFIG_LIBDC1394_INDEV) += libdc1394.o # Objects duplicated from other libraries for shared builds SHLIBOBJS-$(CONFIG_DECKLINK_INDEV) += reverse.o +SHLIBOBJS-$(CONFIG_DECKLINK_OUTDEV) += ccfifo.o # Windows resource file SHLIBOBJS-$(HAVE_GNU_WINDRES) += avdeviceres.o diff --git a/libavdevice/ccfifo.c b/libavdevice/ccfifo.c new file mode 100644 index 0000000..9007094 --- /dev/null +++ b/libavdevice/ccfifo.c @@ -0,0 +1,24 @@ +/* + * CEA-708 Closed Captioning FIFO + * Copyright (c) 2023 LTN Global Communications + * + * Author: Devin Heitmueller + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavfilter/ccfifo.c" diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 088e165..0d33f94 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -31,6 +31,7 @@ extern "C" { #include "libavcodec/packet_internal.h" +#include "libavfilter/ccfifo.h" } #include "libavutil/thread.h" #include "decklink_common_c.h" @@ -112,6 +113,8 @@ struct decklink_ctx { /* Capture buffer queue */ AVPacketQueue queue; + AVCCFifo *cc_fifo; ///< closed captions + /* Streams present */ int audio; int video; diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index 9f1a8df..22aa80b 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -326,6 +326,25 @@ static int create_s337_payload(AVPacket *pkt, uint8_t **outbuf, int *outsize) return 0; } +static int decklink_setup_subtitle(AVFormatContext *avctx, AVStream *st) +{ + int ret = -1; + + switch(st->codecpar->codec_id) { +#if CONFIG_LIBKLVANC + case AV_CODEC_ID_EIA_608: + /* No special setup required */ + ret = 0; + break; +#endif + default: + av_log(avctx, AV_LOG_ERROR, "Unsupported subtitle codec specified\n"); + break; + } + + return ret; +} + av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) { struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; @@ -352,6 +371,7 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) klvanc_context_destroy(ctx->vanc_ctx); #endif + ff_ccfifo_freep(&ctx->cc_fifo); av_freep(&cctx->ctx); return 0; @@ -503,6 +523,21 @@ out: free(afd_words); } +/* Parse any EIA-608 subtitles sitting on the queue, and write packet side data + that will later be handled by construct_cc... */ +static void parse_608subs(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt) +{ + size_t cc_size = ff_ccfifo_getoutputsize(ctx->cc_fifo); + uint8_t *cc_data; + + if (!ff_ccfifo_ccdetected(ctx->cc_fifo)) + return; + + cc_data = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, cc_size); + if (cc_data) + ff_ccfifo_injectbytes(ctx->cc_fifo, cc_data, cc_size); +} + static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt, decklink_frame *frame, AVStream *st) @@ -513,6 +548,7 @@ static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx * if (!ctx->supports_vanc) return 0; + parse_608subs(avctx, ctx, pkt); construct_cc(avctx, ctx, pkt, &vanc_lines); construct_afd(avctx, ctx, pkt, &vanc_lines, st); @@ -704,6 +740,16 @@ static int decklink_write_audio_packet(AVFormatContext *avctx, AVPacket *pkt) return ret; } +static int decklink_write_subtitle_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + ff_ccfifo_extractbytes(ctx->cc_fifo, pkt->data, pkt->size); + + return 0; +} + extern "C" { av_cold int ff_decklink_write_header(AVFormatContext *avctx) @@ -768,12 +814,29 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) } else if (c->codec_type == AVMEDIA_TYPE_VIDEO) { if (decklink_setup_video(avctx, st)) goto error; + } else if (c->codec_type == AVMEDIA_TYPE_SUBTITLE) { + if (decklink_setup_subtitle(avctx, st)) + goto error; } else { av_log(avctx, AV_LOG_ERROR, "Unsupported stream type.\n"); goto error; } } + for (n = 0; n < avctx->nb_streams; n++) { + AVStream *st = avctx->streams[n]; + AVCodecParameters *c = st->codecpar; + + if(c->codec_type == AVMEDIA_TYPE_SUBTITLE) + avpriv_set_pts_info(st, 64, ctx->bmd_tb_num, ctx->bmd_tb_den); + } + + if (!(ctx->cc_fifo = ff_ccfifo_alloc(av_make_q(ctx->bmd_tb_den, ctx->bmd_tb_num), avctx))) { + av_log(ctx, AV_LOG_ERROR, "Failure to setup CC FIFO queue\n"); + ret = AVERROR(ENOMEM); + goto error; + } + return 0; error: @@ -789,6 +852,8 @@ int ff_decklink_write_packet(AVFormatContext *avctx, AVPacket *pkt) return decklink_write_video_packet(avctx, pkt); else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) return decklink_write_audio_packet(avctx, pkt); + else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) + return decklink_write_subtitle_packet(avctx, pkt); return AVERROR(EIO); } diff --git a/libavdevice/decklink_enc_c.c b/libavdevice/decklink_enc_c.c index f7e3150..0a3984b 100644 --- a/libavdevice/decklink_enc_c.c +++ b/libavdevice/decklink_enc_c.c @@ -77,7 +77,7 @@ const FFOutputFormat ff_decklink_muxer = { .p.long_name = NULL_IF_CONFIG_SMALL("Blackmagic DeckLink output"), .p.audio_codec = AV_CODEC_ID_PCM_S16LE, .p.video_codec = AV_CODEC_ID_WRAPPED_AVFRAME, - .p.subtitle_codec = AV_CODEC_ID_NONE, + .p.subtitle_codec = AV_CODEC_ID_EIA_608, .p.flags = AVFMT_NOFILE, .p.priv_class = &decklink_muxer_class, .get_device_list = ff_decklink_list_output_devices,