From patchwork Wed Sep 11 18:56:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 15026 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 441D444843B for ; Wed, 11 Sep 2019 21:56:27 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 307DC687FA5; Wed, 11 Sep 2019 21:56:27 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1411D68098D for ; Wed, 11 Sep 2019 21:56:20 +0300 (EEST) Received: by mail-ot1-f45.google.com with SMTP id 41so19808459oti.12 for ; Wed, 11 Sep 2019 11:56:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=p7JsHFtQbJREWchjrYcrnp8TUbAvru/OdbfHn4JN83A=; b=nZzmlFxMLvBKf0f+oMzVY4wcsOIdg772L+41FxfEdWm4sxbZfFcM8U36bw3iFj0syi aBSU5hc8BxDfnTh7jim33l9KWj6nIl4ZbP5qhOl5NxEqfsAsRKyaENvSXzgyButJpWVP xMPO5CwJQQ6VZWkbjfA8019HjXE/GWMATazQtSeER1XH9OnhBRElllFr95SM//Ro3r/B MmxtIX3S91h3dQ+08tucC3IzjTeeXWBiK5wspyR5pQIEmHvNf7XycZvaNl7CS3AEdRYU ozeDjxk9yASsWilmErSl3NP0rgQIR+KFRh7roYCyRzL7Vw29Grf9SMGx1mp/MQJ0hdn4 dtKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=p7JsHFtQbJREWchjrYcrnp8TUbAvru/OdbfHn4JN83A=; b=rxZ9CspMHtxni5a5yaofAXUpSKDr8EB4srtQZZXrp/im3rr+LfvC3QLebhlSImqFta qlTokN9G9UjRzFGBePzC9Xee8cUSwCeyZQD4/3zLMEbsnSWpG08VnMdWqX5rEp49evL5 KkmIIOnlbORuk/cqEjNTYM+f00O3fYcbmc2TeLnYMzaf1llg1jkzAGaOYrTo2YsUqiVb 4rypo3SVniT1mEWQ7Gg7cMKDVET3Q5ZFxSgeQOFBZfJLhlM8xU+6APj2FsQnX/+k/tq7 f3w6rYHGO/gp8ffChAsRo4xOs9s2Rr1dJCbaxx7u0cJlOJ7GvBF9C72fUf16biaudMjJ lDQw== X-Gm-Message-State: APjAAAV8RHwssY8ZBjDhoOr4JjdJhCGkPVJjvtlOQVB4cE+ldyW1Zrnp m1xQNpSkoz5ln9shQHpJS3LrNXsJ3z5b5Q== X-Google-Smtp-Source: APXvYqyEemJm7nRxcgughprWQ/hDu6IfVmdU/V4n6VGAh+NYvzR1mfXAj39VGpgjSO6Ouko5R5M15A== X-Received: by 2002:a9d:6206:: with SMTP id g6mr25511636otj.362.1568228178048; Wed, 11 Sep 2019 11:56:18 -0700 (PDT) Received: from tmm1-imac.lan (ip184-189-221-177.sb.sd.cox.net. [184.189.221.177]) by smtp.gmail.com with ESMTPSA id 34sm1487924otf.55.2019.09.11.11.56.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 11 Sep 2019 11:56:17 -0700 (PDT) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Wed, 11 Sep 2019 11:56:08 -0700 Message-Id: <20190911185610.87081-2-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190911185610.87081-1-ffmpeg@tmm1.net> References: <20190911185610.87081-1-ffmpeg@tmm1.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/4] h264_metadata: Add support for A/53 closed captions X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: sw@jkqxz.net, andreas.rheinhardt@gmail.com Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Mark Thompson Allows insertion (from side data), extraction (to side data), and removal of closed captions in SEI messages. --- libavcodec/Makefile | 2 +- libavcodec/h264_metadata_bsf.c | 133 +++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 1 deletion(-) diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 6bc4383c8f..b10064776c 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1079,7 +1079,7 @@ OBJS-$(CONFIG_EAC3_CORE_BSF) += eac3_core_bsf.o OBJS-$(CONFIG_EXTRACT_EXTRADATA_BSF) += extract_extradata_bsf.o \ av1_parse.o h2645_parse.o OBJS-$(CONFIG_FILTER_UNITS_BSF) += filter_units_bsf.o -OBJS-$(CONFIG_H264_METADATA_BSF) += h264_metadata_bsf.o h264_levels.o +OBJS-$(CONFIG_H264_METADATA_BSF) += h264_metadata_bsf.o h264_levels.o cbs_misc.o OBJS-$(CONFIG_H264_MP4TOANNEXB_BSF) += h264_mp4toannexb_bsf.o OBJS-$(CONFIG_H264_REDUNDANT_PPS_BSF) += h264_redundant_pps_bsf.o OBJS-$(CONFIG_HAPQA_EXTRACT_BSF) += hapqa_extract_bsf.o hap.o diff --git a/libavcodec/h264_metadata_bsf.c b/libavcodec/h264_metadata_bsf.c index 5de74be9d6..65c7b7b55e 100644 --- a/libavcodec/h264_metadata_bsf.c +++ b/libavcodec/h264_metadata_bsf.c @@ -24,6 +24,7 @@ #include "bsf.h" #include "cbs.h" #include "cbs_h264.h" +#include "cbs_misc.h" #include "h264.h" #include "h264_levels.h" #include "h264_sei.h" @@ -84,6 +85,7 @@ typedef struct H264MetadataContext { int flip; int level; + int a53_cc; } H264MetadataContext; @@ -281,6 +283,8 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt) CodedBitstreamFragment *au = &ctx->access_unit; int err, i, j, has_sps; H264RawAUD aud; + uint8_t *a53_side_data = NULL; + size_t a53_side_data_size = 0; err = ff_bsf_get_packet_ref(bsf, pkt); if (err < 0) @@ -556,6 +560,122 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt) } } + if (ctx->a53_cc == INSERT) { + uint8_t *data; + int size; + + data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size); + if (data) { + A53UserData a53_ud; + + err = ff_cbs_read_a53_cc_side_data(ctx->cbc, &a53_ud, + data, size); + if (err < 0) { + av_log(bsf, AV_LOG_WARNING, "Invalid A/53 closed captions " + "in packet side data dropped.\n"); + } else { + H264RawSEIPayload payload = { + .payload_type = H264_SEI_TYPE_USER_DATA_REGISTERED, + }; + H264RawSEIUserDataRegistered *udr = + &payload.payload.user_data_registered; + size_t size = 9 + 3 * a53_ud.atsc.cc_data.cc_count; + + udr->data_ref = av_buffer_alloc(2 + size); + if (!udr->data_ref) { + err = AVERROR(ENOMEM); + goto fail; + } + udr->data = udr->data_ref->data; + + udr->itu_t_t35_country_code = 181; + AV_WB16(udr->data, 49); // provider_code + + err = ff_cbs_write_a53_user_data(ctx->cbc, udr->data + 2, + &size, &a53_ud); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write " + "A/53 user data.\n"); + av_buffer_unref(&udr->data_ref); + goto fail; + } + udr->data_length = size + 2; + + err = ff_cbs_h264_add_sei_message(ctx->cbc, au, &payload); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to add A/53 user data " + "SEI message to access unit.\n"); + av_buffer_unref(&udr->data_ref); + goto fail; + } + } + } + + } else if (ctx->a53_cc == REMOVE || ctx->a53_cc == EXTRACT) { + for (i = 0; i < au->nb_units; i++) { + H264RawSEI *sei; + if (au->units[i].type != H264_NAL_SEI) + continue; + sei = au->units[i].content; + + for (j = 0; j < sei->payload_count; j++) { + H264RawSEIUserDataRegistered *udr; + A53UserData a53_ud; + + if (sei->payload[j].payload_type != + H264_SEI_TYPE_USER_DATA_REGISTERED) + continue; + udr = &sei->payload[j].payload.user_data_registered; + if (udr->data_length < 6) { + // Can't be relevant. + continue; + } + + err = ff_cbs_read_a53_user_data(ctx->cbc, &a53_ud, + udr->data + 2, + udr->data_length - 2); + if (err < 0) { + // Invalid or something completely different. + continue; + } + if (a53_ud.user_identifier != A53_USER_IDENTIFIER_ATSC || + a53_ud.atsc.user_data_type_code != + A53_USER_DATA_TYPE_CODE_CC_DATA) { + // Valid but something else (e.g. AFD). + continue; + } + + if (ctx->a53_cc == REMOVE) { + ff_cbs_h264_delete_sei_message(ctx->cbc, au, + &au->units[i], j); + --i; + break; + } else if(ctx->a53_cc == EXTRACT) { + err = ff_cbs_write_a53_cc_side_data(ctx->cbc, + &a53_side_data, + &a53_side_data_size, + &a53_ud); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write " + "A/53 user data for packet side data.\n"); + goto fail; + } + + if (a53_side_data) { + err = av_packet_add_side_data(pkt, AV_PKT_DATA_A53_CC, + a53_side_data, a53_side_data_size); + if (err) { + av_log(bsf, AV_LOG_ERROR, "Failed to attach extracted A/53 " + "side data to packet.\n"); + goto fail; + } + a53_side_data = NULL; + } + } + } + } + } + err = ff_cbs_write_packet(ctx->cbc, pkt, au); if (err < 0) { av_log(bsf, AV_LOG_ERROR, "Failed to write packet.\n"); @@ -567,6 +687,7 @@ static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt) err = 0; fail: ff_cbs_fragment_reset(ctx->cbc, au); + av_freep(&a53_side_data); if (err < 0) av_packet_unref(pkt); @@ -742,6 +863,18 @@ static const AVOption h264_metadata_options[] = { { LEVEL("6.2", 62) }, #undef LEVEL + { "a53_cc", "A/53 Closed Captions in SEI NAL units", + OFFSET(a53_cc), AV_OPT_TYPE_INT, + { .i64 = PASS }, PASS, EXTRACT, FLAGS, "a53_cc" }, + { "pass", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = PASS }, .flags = FLAGS, .unit = "a53_cc" }, + { "insert", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = INSERT }, .flags = FLAGS, .unit = "a53_cc" }, + { "remove", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = REMOVE }, .flags = FLAGS, .unit = "a53_cc" }, + { "extract", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = EXTRACT }, .flags = FLAGS, .unit = "a53_cc" }, + { NULL } };