[FFmpeg-devel,2/4] h264_metadata: Add support for A/53 closed captions

Submitted by Aman Gupta on Sept. 11, 2019, 6:56 p.m.

Details

Message ID 20190911185610.87081-2-ffmpeg@tmm1.net
State New
Headers show

Commit Message

Aman Gupta Sept. 11, 2019, 6:56 p.m.
From: Mark Thompson <sw@jkqxz.net>

Allows insertion (from side data), extraction (to side data), and removal
of closed captions in SEI messages.
---
 libavcodec/Makefile            |   2 +-
 libavcodec/h264_metadata_bsf.c | 133 +++++++++++++++++++++++++++++++++
 2 files changed, 134 insertions(+), 1 deletion(-)

Comments

Carl Eugen Hoyos Sept. 11, 2019, 8:41 p.m.
Am Mi., 11. Sept. 2019 um 20:56 Uhr schrieb Aman Gupta <ffmpeg@tmm1.net>:
>
> From: Mark Thompson <sw@jkqxz.net>
>
> Allows insertion (from side data), extraction (to side data), and removal
> of closed captions in SEI messages.

Please mention ticket #5283.

Carl Eugen
Michael Niedermayer Sept. 13, 2019, 6:28 p.m.
On Wed, Sep 11, 2019 at 11:56:08AM -0700, Aman Gupta wrote:
> From: Mark Thompson <sw@jkqxz.net>
> 
> Allows insertion (from side data), extraction (to side data), and removal
> of closed captions in SEI messages.
> ---
>  libavcodec/Makefile            |   2 +-
>  libavcodec/h264_metadata_bsf.c | 133 +++++++++++++++++++++++++++++++++
>  2 files changed, 134 insertions(+), 1 deletion(-)

breaks build on mips

CC	libavcodec/cbs_misc.o
In file included from src/libavcodec/cbs_misc.c:25:
src/libavcodec/cbs_misc.h:73: warning: declaration does not declare anything
src/libavcodec/cbs_misc.h:86: warning: declaration does not declare anything
In file included from src/libavcodec/cbs_misc.c:57:
src/libavcodec/cbs_misc_syntax_template.c: In function ‘cbs_misc_read_a53_atsc_user_data’:
src/libavcodec/cbs_misc_syntax_template.c:102: error: ‘A53ATSCUserData’ has no member named ‘cc_data’
src/libavcodec/cbs_misc_syntax_template.c:104: error: ‘A53ATSCUserData’ has no member named ‘bar_data’
src/libavcodec/cbs_misc_syntax_template.c: In function ‘cbs_misc_read_a53_user_data’:
src/libavcodec/cbs_misc_syntax_template.c:140: error: ‘A53UserData’ has no member named ‘atsc’
src/libavcodec/cbs_misc_syntax_template.c:142: error: ‘A53UserData’ has no member named ‘afd’
In file included from src/libavcodec/cbs_misc.c:82:
src/libavcodec/cbs_misc_syntax_template.c: In function ‘cbs_misc_write_a53_atsc_user_data’:
src/libavcodec/cbs_misc_syntax_template.c:102: error: ‘A53ATSCUserData’ has no member named ‘cc_data’
src/libavcodec/cbs_misc_syntax_template.c:104: error: ‘A53ATSCUserData’ has no member named ‘bar_data’
src/libavcodec/cbs_misc_syntax_template.c: In function ‘cbs_misc_write_a53_user_data’:
src/libavcodec/cbs_misc_syntax_template.c:140: error: ‘A53UserData’ has no member named ‘atsc’
src/libavcodec/cbs_misc_syntax_template.c:142: error: ‘A53UserData’ has no member named ‘afd’
src/libavcodec/cbs_misc.c: In function ‘ff_cbs_read_a53_cc_side_data’:
src/libavcodec/cbs_misc.c:153: error: unknown field ‘atsc’ specified in initializer
src/libavcodec/cbs_misc.c:153: error: extra brace group at end of initializer
src/libavcodec/cbs_misc.c:153: error: (near initialization for ‘(anonymous)’)
src/libavcodec/cbs_misc.c:156: error: extra brace group at end of initializer
src/libavcodec/cbs_misc.c:156: error: (near initialization for ‘(anonymous)’)
src/libavcodec/cbs_misc.c:165: warning: excess elements in struct initializer
src/libavcodec/cbs_misc.c:165: warning: (near initialization for ‘(anonymous)’)
src/libavcodec/cbs_misc.c:167: error: ‘A53UserData’ has no member named ‘atsc’
src/libavcodec/cbs_misc.c: In function ‘ff_cbs_write_a53_cc_side_data’:
src/libavcodec/cbs_misc.c:193: error: ‘A53UserData’ has no member named ‘atsc’
src/libavcodec/cbs_misc.c:196: error: ‘A53UserData’ has no member named ‘atsc’
make: *** [libavcodec/cbs_misc.o] Error 1

[...]
Carl Eugen Hoyos Sept. 13, 2019, 7:29 p.m.
Am Fr., 13. Sept. 2019 um 20:28 Uhr schrieb Michael Niedermayer
<michael@niedermayer.cc>:
>
> On Wed, Sep 11, 2019 at 11:56:08AM -0700, Aman Gupta wrote:
> > From: Mark Thompson <sw@jkqxz.net>
> >
> > Allows insertion (from side data), extraction (to side data), and removal
> > of closed captions in SEI messages.
> > ---
> >  libavcodec/Makefile            |   2 +-
> >  libavcodec/h264_metadata_bsf.c | 133 +++++++++++++++++++++++++++++++++
> >  2 files changed, 134 insertions(+), 1 deletion(-)
>
> breaks build on mips
>
> CC      libavcodec/cbs_misc.o
> In file included from src/libavcodec/cbs_misc.c:25:
> src/libavcodec/cbs_misc.h:73: warning: declaration does not declare anything
> src/libavcodec/cbs_misc.h:86: warning: declaration does not declare anything

I can confirm that gcc-4.4.6 (which passes fate here with current FFmpeg)
does not support these (anonymous?) unions.

Carl Eugen

Patch hide | download patch | download mbox

diff --git a/libavcodec/Makefile b/libavcodec/Makefile
index 6bc4383c8f..b10064776c 100644
--- a/libavcodec/Makefile
+++ b/libavcodec/Makefile
@@ -1079,7 +1079,7 @@  OBJS-$(CONFIG_EAC3_CORE_BSF)              += eac3_core_bsf.o
 OBJS-$(CONFIG_EXTRACT_EXTRADATA_BSF)      += extract_extradata_bsf.o    \
                                              av1_parse.o h2645_parse.o
 OBJS-$(CONFIG_FILTER_UNITS_BSF)           += filter_units_bsf.o
-OBJS-$(CONFIG_H264_METADATA_BSF)          += h264_metadata_bsf.o h264_levels.o
+OBJS-$(CONFIG_H264_METADATA_BSF)          += h264_metadata_bsf.o h264_levels.o cbs_misc.o
 OBJS-$(CONFIG_H264_MP4TOANNEXB_BSF)       += h264_mp4toannexb_bsf.o
 OBJS-$(CONFIG_H264_REDUNDANT_PPS_BSF)     += h264_redundant_pps_bsf.o
 OBJS-$(CONFIG_HAPQA_EXTRACT_BSF)          += hapqa_extract_bsf.o hap.o
diff --git a/libavcodec/h264_metadata_bsf.c b/libavcodec/h264_metadata_bsf.c
index 5de74be9d6..65c7b7b55e 100644
--- a/libavcodec/h264_metadata_bsf.c
+++ b/libavcodec/h264_metadata_bsf.c
@@ -24,6 +24,7 @@ 
 #include "bsf.h"
 #include "cbs.h"
 #include "cbs_h264.h"
+#include "cbs_misc.h"
 #include "h264.h"
 #include "h264_levels.h"
 #include "h264_sei.h"
@@ -84,6 +85,7 @@  typedef struct H264MetadataContext {
     int flip;
 
     int level;
+    int a53_cc;
 } H264MetadataContext;
 
 
@@ -281,6 +283,8 @@  static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt)
     CodedBitstreamFragment *au = &ctx->access_unit;
     int err, i, j, has_sps;
     H264RawAUD aud;
+    uint8_t *a53_side_data = NULL;
+    size_t a53_side_data_size = 0;
 
     err = ff_bsf_get_packet_ref(bsf, pkt);
     if (err < 0)
@@ -556,6 +560,122 @@  static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt)
         }
     }
 
+    if (ctx->a53_cc == INSERT) {
+        uint8_t *data;
+        int size;
+
+        data = av_packet_get_side_data(pkt, AV_PKT_DATA_A53_CC, &size);
+        if (data) {
+            A53UserData a53_ud;
+
+            err = ff_cbs_read_a53_cc_side_data(ctx->cbc, &a53_ud,
+                                               data, size);
+            if (err < 0) {
+                av_log(bsf, AV_LOG_WARNING, "Invalid A/53 closed captions "
+                       "in packet side data dropped.\n");
+            } else {
+                H264RawSEIPayload payload = {
+                    .payload_type = H264_SEI_TYPE_USER_DATA_REGISTERED,
+                };
+                H264RawSEIUserDataRegistered *udr =
+                    &payload.payload.user_data_registered;
+                size_t size = 9 + 3 * a53_ud.atsc.cc_data.cc_count;
+
+                udr->data_ref = av_buffer_alloc(2 + size);
+                if (!udr->data_ref) {
+                    err = AVERROR(ENOMEM);
+                    goto fail;
+                }
+                udr->data = udr->data_ref->data;
+
+                udr->itu_t_t35_country_code = 181;
+                AV_WB16(udr->data, 49); // provider_code
+
+                err = ff_cbs_write_a53_user_data(ctx->cbc, udr->data + 2,
+                                                 &size, &a53_ud);
+                if (err < 0) {
+                    av_log(bsf, AV_LOG_ERROR, "Failed to write "
+                           "A/53 user data.\n");
+                    av_buffer_unref(&udr->data_ref);
+                    goto fail;
+                }
+                udr->data_length = size + 2;
+
+                err = ff_cbs_h264_add_sei_message(ctx->cbc, au, &payload);
+                if (err < 0) {
+                    av_log(bsf, AV_LOG_ERROR, "Failed to add A/53 user data "
+                           "SEI message to access unit.\n");
+                    av_buffer_unref(&udr->data_ref);
+                    goto fail;
+                }
+            }
+        }
+
+    } else if (ctx->a53_cc == REMOVE || ctx->a53_cc == EXTRACT) {
+        for (i = 0; i < au->nb_units; i++) {
+            H264RawSEI *sei;
+            if (au->units[i].type != H264_NAL_SEI)
+                continue;
+            sei = au->units[i].content;
+
+            for (j = 0; j < sei->payload_count; j++) {
+                H264RawSEIUserDataRegistered *udr;
+                A53UserData a53_ud;
+
+                if (sei->payload[j].payload_type !=
+                    H264_SEI_TYPE_USER_DATA_REGISTERED)
+                    continue;
+                udr = &sei->payload[j].payload.user_data_registered;
+                if (udr->data_length < 6) {
+                    // Can't be relevant.
+                    continue;
+                }
+
+                err = ff_cbs_read_a53_user_data(ctx->cbc, &a53_ud,
+                                                udr->data + 2,
+                                                udr->data_length - 2);
+                if (err < 0) {
+                    // Invalid or something completely different.
+                    continue;
+                }
+                if (a53_ud.user_identifier != A53_USER_IDENTIFIER_ATSC ||
+                    a53_ud.atsc.user_data_type_code !=
+                        A53_USER_DATA_TYPE_CODE_CC_DATA) {
+                    // Valid but something else (e.g. AFD).
+                    continue;
+                }
+
+                if (ctx->a53_cc == REMOVE) {
+                    ff_cbs_h264_delete_sei_message(ctx->cbc, au,
+                                                         &au->units[i], j);
+                    --i;
+                    break;
+                } else if(ctx->a53_cc == EXTRACT) {
+                    err = ff_cbs_write_a53_cc_side_data(ctx->cbc,
+                                                        &a53_side_data,
+                                                        &a53_side_data_size,
+                                                        &a53_ud);
+                    if (err < 0) {
+                        av_log(bsf, AV_LOG_ERROR, "Failed to write "
+                               "A/53 user data for packet side data.\n");
+                        goto fail;
+                    }
+
+                    if (a53_side_data) {
+                        err = av_packet_add_side_data(pkt, AV_PKT_DATA_A53_CC,
+                                                      a53_side_data, a53_side_data_size);
+                        if (err) {
+                            av_log(bsf, AV_LOG_ERROR, "Failed to attach extracted A/53 "
+                                   "side data to packet.\n");
+                            goto fail;
+                        }
+                        a53_side_data = NULL;
+                    }
+                }
+            }
+        }
+    }
+
     err = ff_cbs_write_packet(ctx->cbc, pkt, au);
     if (err < 0) {
         av_log(bsf, AV_LOG_ERROR, "Failed to write packet.\n");
@@ -567,6 +687,7 @@  static int h264_metadata_filter(AVBSFContext *bsf, AVPacket *pkt)
     err = 0;
 fail:
     ff_cbs_fragment_reset(ctx->cbc, au);
+    av_freep(&a53_side_data);
 
     if (err < 0)
         av_packet_unref(pkt);
@@ -742,6 +863,18 @@  static const AVOption h264_metadata_options[] = {
     { LEVEL("6.2", 62) },
 #undef LEVEL
 
+    { "a53_cc", "A/53 Closed Captions in SEI NAL units",
+        OFFSET(a53_cc), AV_OPT_TYPE_INT,
+        { .i64 = PASS }, PASS, EXTRACT, FLAGS, "a53_cc" },
+    { "pass",    NULL, 0, AV_OPT_TYPE_CONST,
+        { .i64 = PASS    }, .flags = FLAGS, .unit = "a53_cc" },
+    { "insert",  NULL, 0, AV_OPT_TYPE_CONST,
+        { .i64 = INSERT  }, .flags = FLAGS, .unit = "a53_cc" },
+    { "remove",  NULL, 0, AV_OPT_TYPE_CONST,
+        { .i64 = REMOVE  }, .flags = FLAGS, .unit = "a53_cc" },
+    { "extract", NULL, 0, AV_OPT_TYPE_CONST,
+        { .i64 = EXTRACT }, .flags = FLAGS, .unit = "a53_cc" },
+
     { NULL }
 };