From patchwork Sat Feb 4 00:41:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 40242 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5494:b0:bf:7b3a:fd32 with SMTP id i20csp1617172pzk; Fri, 3 Feb 2023 16:42:49 -0800 (PST) X-Google-Smtp-Source: AK7set8x3rxpg7wUUVBPzKOo3HIrSBPgo6zUyI3vNcvsp3CIpfSEvCGx7sHmG7lTOE7AYG//dWWw X-Received: by 2002:a17:906:8586:b0:889:1eb1:7517 with SMTP id v6-20020a170906858600b008891eb17517mr12173107ejx.30.1675471369204; Fri, 03 Feb 2023 16:42:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675471369; cv=none; d=google.com; s=arc-20160816; b=R29KQYKzwhsZLaxBd1+lYMANTTvXbx29xkt/r6bDthqs88Ey/4J3gRsRtLgfr0DDcI Syvpl+Ls+wwiHtvvFIT0xvi6lQsWFkoKJDsfWdPLjyQFoCxBfuPxzMuva9GgfB3WhHsk S0rWm+qs7GxayVvTo/6uRgEFFBE7CMmTtp6ADSZcq2KdYzhG+y56Cjivhzpp/gYOQl3Z ZrBpEbuVgDEqDcvZL87aArsZD0zAgAYNOIEgXdmLYzA4FuCZOVC6mIrW10QNzjf8eHae ULKSO0rSM9Q+K489aQ57xqxt3KGFK3cfrV6XSpTvyzsA7elNUaStWRYa5jVDLhx/Cx+6 uZHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:to:mime-version:fcc:date:references :in-reply-to:message-id:from:dkim-signature:delivered-to; bh=Q2zp/KBDW8PkRmMFoNZRdR538tGeudXpw7zDNZcv3JA=; b=CZbCuAutxVbJOVD0CXMcZM3DF1HNm/o/u4wx/CJYKfrHwDR3ruFUYsxjnanEzOZdVp ghoOIPk2XDm92BEzwPGgmmUZTlVEjAQ8yDq4ualp38huoyVX9oDTiq1VUq1WfMZ2Zr12 NfVEOZw5LBXpS3l3pOvQQf6apdju1YtmVIpWWiYfEDav6NPZkiSqt7yEGIncWPioF7/6 VQednV4cMFMQRlntlQKBl2x+frpGy51P1r+3mHto/uX9H9NK2sK4ERj8H/5ISCRk2IPc Mx4CSVKd5HlguddXvaRDsFCnQbI+PNv5PQxnqHqC7JdrkYriRNxLW/Vmwxm+pJ1thr0d 3kNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=eB5ELDnj; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 14-20020a170906024e00b0087879f8c291si5272078ejl.703.2023.02.03.16.42.48; Fri, 03 Feb 2023 16:42:49 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=eB5ELDnj; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 12D3A68BEBF; Sat, 4 Feb 2023 02:42:10 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4C98868BE88 for ; Sat, 4 Feb 2023 02:42:01 +0200 (EET) Received: by mail-pg1-f169.google.com with SMTP id 7so4763923pgh.7 for ; Fri, 03 Feb 2023 16:42:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:from:to:cc:subject:date :message-id:reply-to; bh=aNEtYWwNlxyEBV/qVhmAs8GEVMAR28imFQr1nK2kqvY=; b=eB5ELDnjLFABGL/kJVC8ZH1B2bwMv2QqTBjjtLjfNdL5DgbGxioMjAdkl4MPUz+Rp3 g+9PVzNN5bN4XHKSgb40D6Wd4tXIde7bKHRcx6V41ITnKWMkI7w8Zy5GDh/CRZfTDwEZ yehSiuR81UJCw9kLQQzRzvUeWSzkjC5WWKS/4fLwfPJejrh/xUEQeXLdE/Fa9pBuIzUm 4tn5zqDO3lGl1Hbu2a5jA9z50CZJKgXVFszTJ1bhs0lnHSmExTqJQs7iAeYdtyJiqJI1 JfNEs2w8QCuVF6E3Mf7bvzWLx3wJJSW6sg1Ei5L285ZEqlnDY8gaEUa+9OsmScM6dJnU tTSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date :references:in-reply-to:message-id:from:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=aNEtYWwNlxyEBV/qVhmAs8GEVMAR28imFQr1nK2kqvY=; b=171Po4iQ4ANvosz+zaKJ58gF4oCI89GRF+NmSSL6ZBm/iNuLA+edVuTBhZo9SsxN8L Ue7Lbc+ZREUM+sTVlr22k1tMOsuY6NEhNMb44hfjeIDvPfbsqeaukvXYqbK2fEzCG+Op l5JhjfRXk7zZNQP7qGVHtL8GaNag/o/McYc7BwG9TqY8rJ9G+b0CHHcLlbgFKF+3SZLX oJ5uTKz2JF3c7JX69RQ5EGlxztpKm3+DvEDVCwzckHGs5lBrJSkjJGva9H6JMsYFI7gb +1rgHYxAKTNtGFPyQR9nL/qERbD5sbS/gSLEOsUod0b6xIF1w5njp9UpKz3cq4adKbZr SHvg== X-Gm-Message-State: AO0yUKXqAJ1RbVeV/zywUgpZlyAISYpHpVwjJas/Va3NQn3Npw0riFi4 ziibb76RCVZ/vwWQMaP8wDRBbtQgY2U= X-Received: by 2002:a05:6a00:2911:b0:58b:c1a1:4006 with SMTP id cg17-20020a056a00291100b0058bc1a14006mr10497969pfb.18.1675471319666; Fri, 03 Feb 2023 16:41:59 -0800 (PST) Received: from [127.0.0.1] (master.gitmailbox.com. [34.83.118.50]) by smtp.gmail.com with ESMTPSA id x11-20020aa793ab000000b0058bba6f06c5sm2432872pff.8.2023.02.03.16.41.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Feb 2023 16:41:59 -0800 (PST) From: Aman Karmani X-Google-Original-From: Aman Karmani Message-Id: In-Reply-To: References: Date: Sat, 04 Feb 2023 00:41:54 +0000 Fcc: Sent MIME-Version: 1.0 To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH 4/5] avcodec/h264_metadata_bsf: add support for a/53 closed captions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Aman Karmani Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 6fTsINEvwxZw From: Aman Karmani Signed-off-by: Aman Karmani --- doc/bitstream_filters.texi | 11 +++ libavcodec/Makefile | 2 +- libavcodec/h264_metadata_bsf.c | 144 +++++++++++++++++++++++++++++++++ 3 files changed, 156 insertions(+), 1 deletion(-) diff --git a/doc/bitstream_filters.texi b/doc/bitstream_filters.texi index c036d37403..3d97f66315 100644 --- a/doc/bitstream_filters.texi +++ b/doc/bitstream_filters.texi @@ -243,6 +243,17 @@ ffmpeg -i hapqa_inputfile.mov -c copy -bsf:v hapqa_extract=texture=alpha -tag:v Modify metadata embedded in an H.264 stream. @table @option +@item a53_cc +Insert or remove registered userdata SEI NAL units containing A/53 closed captions. + +@table @samp +@item pass +@item insert +@item remove +@end table + +Default is pass. + @item aud Insert or remove AUD NAL units in all access units of the stream. diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 2a36a31ff2..9f97c9b5e5 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1212,7 +1212,7 @@ OBJS-$(CONFIG_EXTRACT_EXTRADATA_BSF) += extract_extradata_bsf.o \ av1_parse.o h2645_parse.o OBJS-$(CONFIG_FILTER_UNITS_BSF) += filter_units_bsf.o OBJS-$(CONFIG_H264_METADATA_BSF) += h264_metadata_bsf.o h264_levels.o \ - h2645data.o + h2645data.o cbs_misc.o OBJS-$(CONFIG_H264_MP4TOANNEXB_BSF) += h264_mp4toannexb_bsf.o OBJS-$(CONFIG_H264_REDUNDANT_PPS_BSF) += h264_redundant_pps_bsf.o OBJS-$(CONFIG_HAPQA_EXTRACT_BSF) += hapqa_extract_bsf.o hap.o diff --git a/libavcodec/h264_metadata_bsf.c b/libavcodec/h264_metadata_bsf.c index d318bf0cee..bbe74c3cee 100644 --- a/libavcodec/h264_metadata_bsf.c +++ b/libavcodec/h264_metadata_bsf.c @@ -26,6 +26,8 @@ #include "cbs.h" #include "cbs_bsf.h" #include "cbs_h264.h" +#include "cbs_misc.h" +#include "cbs_sei.h" #include "h264.h" #include "h264_levels.h" #include "h264_sei.h" @@ -81,6 +83,7 @@ typedef struct H264MetadataContext { H264RawSEIDisplayOrientation display_orientation_payload; int level; + int a53_cc; } H264MetadataContext; @@ -463,6 +466,137 @@ static int h264_metadata_handle_display_orientation(AVBSFContext *bsf, return 0; } +static int h264_metadata_handle_a53_cc(AVBSFContext *bsf, AVPacket *pkt, + CodedBitstreamFragment *au) +{ + H264MetadataContext *ctx = bsf->priv_data; + SEIRawMessage *message; + int err; + + message = NULL; + while (ff_cbs_sei_find_message(ctx->common.output, au, + SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35, + &message) == 0) { + SEIRawUserDataRegistered *udr = message->payload; + A53UserData a53_ud; + uint8_t *a53_side_data = NULL; + size_t a53_side_data_size = 0; + + if (udr->data_length < 2) { + // Too short to contain a provider code. + continue; + } + + if (AV_RB16(udr->data) != 0x31) { // provider_code as atsc_provider_code + // Not ATSC. + continue; + } + + // The first two bytes of the message is provider_code so we offset the data by that + err = ff_cbs_read_a53_user_data(ctx->common.output, + &a53_ud, + udr->data + 2, + udr->data_length - 2); + if (err < 0) { + // Invalid or something completely different. + continue; + } + if (a53_ud.user_identifier != A53_USER_IDENTIFIER_ATSC || + a53_ud.atsc.user_data_type_code != + A53_USER_DATA_TYPE_CODE_CC_DATA) { + // Valid but something else (e.g. AFD). + continue; + } + + err = ff_cbs_write_a53_cc_side_data(ctx->common.output, + &a53_side_data, + &a53_side_data_size, + &a53_ud); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write " + "A/53 user data for packet side data.\n"); + return err; + } + + if (a53_side_data) { + err = av_packet_add_side_data(pkt, AV_PKT_DATA_A53_CC, + a53_side_data, a53_side_data_size); + if (err) { + av_log(bsf, AV_LOG_ERROR, "Failed to attach extracted A/53 " + "side data to packet.\n"); + av_freep(&a53_side_data); + return err; + } + + if (ctx->a53_cc == BSF_ELEMENT_REMOVE || + ctx->a53_cc == BSF_ELEMENT_INSERT) { + ff_cbs_sei_delete_message(ctx->common.output, au, message); + + // Reset iteration + message = NULL; + } + } + } + + if (ctx->a53_cc == BSF_ELEMENT_INSERT) { + uint8_t *data; + size_t size; + int offset = 0; + + data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size); + while (size > 0) { + A53UserData a53_ud; + int rsize = FFMIN(93, size); + + err = ff_cbs_read_a53_cc_side_data(ctx->common.output, &a53_ud, + data + offset, rsize); + offset += rsize; + size -= rsize; + if (err < 0) { + av_log(bsf, AV_LOG_WARNING, "Invalid A/53 closed captions " + "in packet side data dropped.\n"); + } else { + AVBufferRef *udr_buf = av_buffer_allocz(sizeof(SEIRawUserDataRegistered)); + SEIRawUserDataRegistered *udr = (SEIRawUserDataRegistered*)udr_buf->data; + size_t size = 9 + 3 * a53_ud.atsc.cc_data.cc_count; + + udr->data_ref = av_buffer_allocz(2 + size); + if (!udr->data_ref) { + return AVERROR(ENOMEM); + } + udr->data = udr->data_ref->data; + udr->data_length = udr->data_ref->size; + + udr->itu_t_t35_country_code = 0xB5; // usa_country_code + AV_WB16(udr->data, 0x31); // provider_code as atsc_provider_code + + err = ff_cbs_write_a53_user_data(ctx->common.output, udr->data + 2, + &size, &a53_ud); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write " + "A/53 user data.\n"); + av_buffer_unref(&udr->data_ref); + av_buffer_unref(&udr_buf); + return err; + } + + err = ff_cbs_sei_add_message(ctx->common.output, au, 1, + SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35, + udr, udr_buf); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to add A/53 user data " + "SEI message to access unit.\n"); + av_buffer_unref(&udr->data_ref); + av_buffer_unref(&udr_buf); + return err; + } + } + } + } + + return 0; +} + static int h264_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, CodedBitstreamFragment *au) { @@ -533,6 +667,12 @@ static int h264_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, return err; } + if (pkt && ctx->a53_cc != BSF_ELEMENT_PASS) { + err = h264_metadata_handle_a53_cc(bsf, pkt, au); + if (err < 0) + return err; + } + if (pkt) ctx->done_first_au = 1; @@ -696,6 +836,10 @@ static const AVOption h264_metadata_options[] = { { LEVEL("6.2", 62) }, #undef LEVEL + BSF_ELEMENT_OPTIONS_PIRE("a53_cc", + "A/53 Closed Captions in SEI NAL units", + a53_cc, FLAGS), + { NULL } };