From patchwork Fri Apr 21 21:16:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devin Heitmueller X-Patchwork-Id: 41296 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4645:b0:e3:3194:9d20 with SMTP id eb5csp1596046pzb; Fri, 21 Apr 2023 13:21:19 -0700 (PDT) X-Google-Smtp-Source: AKy350bXE/kTTBO6kluReYnq1n6yl+IzAk702LsjalXBcMTTaRX10qZwQC3E+3Pkqtct14dRkLMx X-Received: by 2002:a05:6402:d1:b0:506:c238:4067 with SMTP id i17-20020a05640200d100b00506c2384067mr6162778edu.8.1682108479070; Fri, 21 Apr 2023 13:21:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682108479; cv=none; d=google.com; s=arc-20160816; b=rKevqqxYPYZPfmWIqFp+AA7GczeI1QJUCHlSobUasGh2lVSyxBnevtpoFWazAsKh0e XcjzRPkxTHcyF/qSXg6omcw1MhGlg91zWDrHgMUJ6Hp3DMQWTobU61y0l8fDbmQNfMJ+ BShu6ep/nYfJUhrDiArwkZre/Ga//N0tk5gmnMexFoXFNlXjsWOSXyDVkWI1R1/PlF3z R7ajnlFesOu5WwIHQh+1jtkmwIaWwq7SEBSRqgJdazrzT4teNpQd9CIsJggl31Ky/7lq 8XjJc/hImb+ie1Vnivea1L0IxIjRdXa3MVZl4YJFttUvlL2LXNUwG1KDeIhNxWZmPDAN k9Bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to; bh=0RKyIOmylZY5iEiVVJjXhoi26Egqw1ySSjda6EKbcrs=; b=UvHnTxyCt2UqbJr+zPr++SCD5g1RHcIqUhuhkiHgPclK14d/cAOS0JEFhfaDqfEJ4K oln4qCJ4kormixADG4nFyRRVaAwbletf70o4Itp5EDM9X/MgtP/H4StsfzWBKuW46Kdu aMkb7HGT3/o4NcM+8oSHz9b64q4ZwwbOM6IFIqxz/YBOP9mUqgsPqykgiAuiS250ZPX9 6/++/qv1c1XypcPLum0jrlgXHGDsv2v9bnc5aoyzvB6yw4LZPIPy24IVO6yk1Kp8eIR2 CgUkcDj5zIEKhsbpEUyyAnaRT5DKPboeNZKfv+tPOt/bE1U1xdhOU1zntkBWtkUQh/T9 Ei3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=TaS8cdOx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r24-20020aa7d598000000b0050476f7c049si4262337edq.682.2023.04.21.13.21.18; Fri, 21 Apr 2023 13:21:19 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@ltnglobal-com.20221208.gappssmtp.com header.s=20221208 header.b=TaS8cdOx; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id DEA1168BECA; Fri, 21 Apr 2023 23:21:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 7566F68AF28 for ; Fri, 21 Apr 2023 23:21:09 +0300 (EEST) Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-5ef8aaf12bdso10061466d6.3 for ; Fri, 21 Apr 2023 13:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ltnglobal-com.20221208.gappssmtp.com; s=20221208; t=1682108468; x=1684700468; h=message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JNShM7wQeRxZqSKDNx4y6/X2Nq+Ih8U9QGNwD2DZa3I=; b=TaS8cdOxSH1apGHC0LUpS/aCCS75hXLybQl77JZN4GsYWD6oSmOd5s+xZQbN4g+659 WMPh1bVAwv1HqyJkTkolad/5kDQ/4867KK4Gt0WwTEQc0ZWwincG/lKhgFudMSr0EJON cejl4AcpIdu5iuTmGhmHDDN4HkXxdtrjxqXXq1BM13ske966Lgk3TEXk7mZyyo4cHMa1 9QTYDVU8JBYh/KORTR5xS+tvYi/TQBRmdzeAYlStVsZEqc8rHMixnrxE+FsZ4RdlZ9lS ZaconQySf99WIi1Ut+4PcFT0x6lc+BFOnzAzaGG3/lkenswHwsWngPjBEReoc4POFx3o 39jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682108468; x=1684700468; h=message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JNShM7wQeRxZqSKDNx4y6/X2Nq+Ih8U9QGNwD2DZa3I=; b=DtRVOGeo2Svo4AnMGABWkS14WX3A2s0Krrbbv+Q6HX8UbC2VFrf3cUYT6GVF1jXw0F k3/JsiZ5aGfj2Chw4mEPaTWi2m4tFbVgIMeP5mBQ0le6LhQzga3N8pCt/nO1ydUsAWx3 J1jSMpBpHQxEmfPw7I6zYXSWPuhD2Aq72gVW8GQG5z7DWy0Ac+b60mS906If3XqE/Lrs hryNHmKFaH9YDyfOxQ8Gg6xkDd+KtmAUReYH0w6sFGARI7L44wOTqqufm/1R61RZ0a6Y xT6gnGr8wvow0eqFColevHZJ8m6u14axYMU8OAIxJYBNuaVaBBj3P6SVOhbtLE6l0MWk ERzQ== X-Gm-Message-State: AAQBX9dZJ0xUWpi776V4FkZvKfTyzZ7kBTwPSzZTJtT37u0zuK46qDVi G3roTCT58Hvj6VjEA+T8g/BW4u/z64XATB+LJes= X-Received: by 2002:a05:6214:da7:b0:56e:9986:4fa9 with SMTP id h7-20020a0562140da700b0056e99864fa9mr10836987qvh.7.1682108468042; Fri, 21 Apr 2023 13:21:08 -0700 (PDT) Received: from ltnt-nyc-580testdevin.livetimenet.com (pool-71-105-132-214.nycmny.fios.verizon.net. [71.105.132.214]) by smtp.gmail.com with ESMTPSA id x11-20020a0ce0cb000000b005ef6bcbb99fsm1394397qvk.22.2023.04.21.13.21.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Apr 2023 13:21:07 -0700 (PDT) From: Devin Heitmueller X-Google-Original-From: Devin Heitmueller To: ffmpeg-devel@ffmpeg.org Date: Fri, 21 Apr 2023 17:16:45 -0400 Message-Id: <1682111805-31679-1-git-send-email-dheitmueller@ltnglobal.com> X-Mailer: git-send-email 1.8.3.1 Subject: [FFmpeg-devel] [RFC/PATCH] decklink_enc: Add support for playout of EIA-608 caption codec X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Devin Heitmueller MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: QixvejQ20lai Today the decklink SDI output supports putting out captions over VANC when the caption data is packet side data. However some container formats such as MP4 support a dedicated subtitle stream, and we end up receiving packets containing the 608 tuples separate from the video frames. Make use of a simple intermediate queue to stash the 608 tuples, and then attach them to the video frame side data such that it can be inserted into VANC via the subsequent call to construct_cc(). Unlike with packet side data that carries the captions associated with that one frame, individual c608 packets can contain tuples for entire batches of frames, so we need to pace out how quickly they are inserted into the video based on the framerate. Example usage: ./ffmpeg -i file.mp4 -map 0:0 -map 0:1 -map 0:3 -codec:2 copy -vcodec v210 -f decklink 'DeckLink Duo (4)' Note that the user needs to specify -codec copy for the c608 subtitle track, since there is no decoding/encoding happening. The CC fifo code here overlaps with some of the functionality found in the libavfilter/ccfifo.[c/h], but that code can't be referenced as it is currently private to libavfilter. If the ccfifo stuff gets moved to libavutil then the routine here can be simplified. Signed-off-by: Devin Heitmueller --- libavdevice/decklink_common.h | 2 + libavdevice/decklink_enc.cpp | 94 ++++++++++++++++++++++++++++++++++++++++++- libavdevice/decklink_enc_c.c | 2 +- 3 files changed, 95 insertions(+), 3 deletions(-) diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 99afcd4..51ad5da 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -31,6 +31,7 @@ extern "C" { #include "libavcodec/packet_internal.h" +#include "libavutil/fifo.h" } #include "libavutil/thread.h" #include "decklink_common_c.h" @@ -114,6 +115,7 @@ struct decklink_ctx { /* Output VANC queue */ AVPacketQueue vanc_queue; + AVFifo *cc_fifo; /* Streams present */ int audio; diff --git a/libavdevice/decklink_enc.cpp b/libavdevice/decklink_enc.cpp index d7357a0..23f00c8 100644 --- a/libavdevice/decklink_enc.cpp +++ b/libavdevice/decklink_enc.cpp @@ -345,6 +345,30 @@ static int decklink_setup_data(AVFormatContext *avctx, AVStream *st) return ret; } +static int decklink_setup_subtitle(AVFormatContext *avctx, AVStream *st) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + int ret = -1; + + switch(st->codecpar->codec_id) { +#if CONFIG_LIBKLVANC + case AV_CODEC_ID_EIA_608: + if (!ctx->cc_fifo) + ctx->cc_fifo = av_fifo_alloc2(128, 3, 0); + if (ctx->cc_fifo) + ret = 0; + break; +#endif + default: + av_log(avctx, AV_LOG_ERROR, "Unsupported subtitle codec specified\n"); + break; + } + + return ret; +} + av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) { struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; @@ -371,6 +395,7 @@ av_cold int ff_decklink_write_trailer(AVFormatContext *avctx) klvanc_context_destroy(ctx->vanc_ctx); #endif avpacket_queue_end(&ctx->vanc_queue); + av_fifo_freep2(&ctx->cc_fifo); av_freep(&cctx->ctx); @@ -523,6 +548,54 @@ out: free(afd_words); } +/* Parse any EIA-608 subtitles sitting on the queue, and write packet side data + that will later be handled by construct_cc... */ +static void parse_608subs(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt) +{ + int cc_count = 0; + int cc_filled = 0; + + if (!ctx->cc_fifo) + return; + + /* 29.97 and 59.94 are really the only two framerates we care about */ + if (ctx->bmd_tb_den == 30000 && ctx->bmd_tb_num == 1001) { + cc_count = 20; + } else if (ctx->bmd_tb_den == 60000 && ctx->bmd_tb_num == 1001) { + cc_count = 10; + } + if (cc_count > 0) { + uint8_t *cc_buf = av_packet_new_side_data(pkt, AV_PKT_DATA_A53_CC, cc_count * 3); + if (!cc_buf) + return; + int len = av_fifo_can_read(ctx->cc_fifo); + if (len > 0) { + uint8_t buf[3]; + av_fifo_read(ctx->cc_fifo, buf, 1); + av_log(avctx, AV_LOG_ERROR, "Got a 608 packet! size=%d\n", len); + memcpy(cc_buf, buf, 3); + cc_filled++; + if (cc_count == 20 && len > 1) { + /* If we're at 30 FPS, we need to insert a second tuple if one is + available and it's for CC3/CC4) */ + av_fifo_peek(ctx->cc_fifo, buf, 1, 0); + if (buf[0] == 0xfd) { + av_fifo_read(ctx->cc_fifo, buf, 1); + av_log(avctx, AV_LOG_ERROR, "Got a 608 packet! size=%d\n", len); + memcpy(cc_buf, buf, 3); + cc_filled++; + } + } + } + /* Pad out the rest of the field */ + for(int i = cc_filled; i < cc_count; i++) { + cc_buf[i*3] = 0xfa; + cc_buf[i*3+1] = 0x00; + cc_buf[i*3+2] = 0x00; + } + } +} + static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx *ctx, AVPacket *pkt, decklink_frame *frame, AVStream *st) @@ -533,6 +606,7 @@ static int decklink_construct_vanc(AVFormatContext *avctx, struct decklink_ctx * if (!ctx->supports_vanc) return 0; + parse_608subs(avctx, ctx, pkt); construct_cc(avctx, ctx, pkt, &vanc_lines); construct_afd(avctx, ctx, pkt, &vanc_lines, st); @@ -793,6 +867,16 @@ static int decklink_write_data_packet(AVFormatContext *avctx, AVPacket *pkt) return 0; } +static int decklink_write_subtitle_packet(AVFormatContext *avctx, AVPacket *pkt) +{ + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data; + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx; + + av_fifo_write(ctx->cc_fifo, pkt->data, pkt->size / 3); + + return 0; +} + extern "C" { av_cold int ff_decklink_write_header(AVFormatContext *avctx) @@ -860,6 +944,9 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) } else if (c->codec_type == AVMEDIA_TYPE_DATA) { if (decklink_setup_data(avctx, st)) goto error; + } else if (c->codec_type == AVMEDIA_TYPE_SUBTITLE) { + if (decklink_setup_subtitle(avctx, st)) + goto error; } else { av_log(avctx, AV_LOG_ERROR, "Unsupported stream type.\n"); goto error; @@ -870,7 +957,8 @@ av_cold int ff_decklink_write_header(AVFormatContext *avctx) for (n = 0; n < avctx->nb_streams; n++) { AVStream *st = avctx->streams[n]; AVCodecParameters *c = st->codecpar; - if (c->codec_type == AVMEDIA_TYPE_DATA) + if (c->codec_type == AVMEDIA_TYPE_DATA || + c->codec_type == AVMEDIA_TYPE_SUBTITLE) avpriv_set_pts_info(st, 64, ctx->bmd_tb_num, ctx->bmd_tb_den); } avpacket_queue_init (avctx, &ctx->vanc_queue); @@ -890,8 +978,10 @@ int ff_decklink_write_packet(AVFormatContext *avctx, AVPacket *pkt) return decklink_write_video_packet(avctx, pkt); else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) return decklink_write_audio_packet(avctx, pkt); - else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) { + else if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA) return decklink_write_data_packet(avctx, pkt); + else if (st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE) { + return decklink_write_subtitle_packet(avctx, pkt); } return AVERROR(EIO); diff --git a/libavdevice/decklink_enc_c.c b/libavdevice/decklink_enc_c.c index 7aab660..d593cca 100644 --- a/libavdevice/decklink_enc_c.c +++ b/libavdevice/decklink_enc_c.c @@ -78,7 +78,7 @@ const FFOutputFormat ff_decklink_muxer = { .p.long_name = NULL_IF_CONFIG_SMALL("Blackmagic DeckLink output"), .p.audio_codec = AV_CODEC_ID_PCM_S16LE, .p.video_codec = AV_CODEC_ID_WRAPPED_AVFRAME, - .p.subtitle_codec = AV_CODEC_ID_NONE, + .p.subtitle_codec = AV_CODEC_ID_EIA_608, .p.flags = AVFMT_NOFILE, .p.priv_class = &decklink_muxer_class, .get_device_list = ff_decklink_list_output_devices,