[FFmpeg-devel,v4,5/6] vf_ccrepack: Add new filter to repack CEA-708 side data

THis filter can correct certain issues seen from upstream sources
where the cc_count is not properly set or the CEA-608 tuples are
not at the start of the payload as expected.

Make use of the ccfifo to extract and immediately repack the CEA-708
side data, thereby removing any extra padding and ensuring the 608
tuples are at the front of the payload.

Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com>
---
 doc/filters.texi          |  10 +++++
 libavfilter/Makefile      |   1 +
 libavfilter/allfilters.c  |   1 +
 libavfilter/vf_ccrepack.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 112 insertions(+)
 create mode 100644 libavfilter/vf_ccrepack.c

On Fri, Apr 28, 2023 at 11:43 PM Devin Heitmueller <
devin.heitmueller@ltnglobal.com> wrote:

> THis filter can correct certain issues seen from upstream sources
> where the cc_count is not properly set or the CEA-608 tuples are
> not at the start of the payload as expected.
>
> Make use of the ccfifo to extract and immediately repack the CEA-708
> side data, thereby removing any extra padding and ensuring the 608
> tuples are at the front of the payload.
>
> Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com>
> ---
>  doc/filters.texi          |  10 +++++
>  libavfilter/Makefile      |   1 +
>  libavfilter/allfilters.c  |   1 +
>  libavfilter/vf_ccrepack.c | 100
> ++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 112 insertions(+)
>  create mode 100644 libavfilter/vf_ccrepack.c
>
> diff --git a/doc/filters.texi b/doc/filters.texi
> index 5dde799..9f67a00 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -8936,6 +8936,16 @@ Only deinterlace frames marked as interlaced.
>  The default value is @code{all}.
>  @end table
>
> +@section ccrepack
> +
> +Repack CEA-708 closed captioning side data
> +
> +This filter fixes various issues seen with commerical encoders
> +related to upstream malformed CEA-708 payloads, specifically
> +incorrect number of tuples (wrong cc_count for the target FPS),
> +and incorrect ordering of tuples (i.e. the CEA-608 tuples are not at
> +the first entries in the payload).
> +
>  @section cas
>
>  Apply Contrast Adaptive Sharpen filter to video stream.
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index 628ade8..3d0b3ad 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -212,6 +212,7 @@ OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER)         +=
> vf_avgblur_opencl.o opencl.o \
>                                                  opencl/avgblur.o boxblur.o
>  OBJS-$(CONFIG_BWDIF_FILTER)                  += vf_bwdif.o yadif_common.o
>  OBJS-$(CONFIG_CAS_FILTER)                    += vf_cas.o
> +OBJS-$(CONFIG_CCREPACK_FILTER)               += vf_ccrepack.o
>  OBJS-$(CONFIG_CHROMABER_VULKAN_FILTER)       += vf_chromaber_vulkan.o
> vulkan.o vulkan_filter.o
>  OBJS-$(CONFIG_CHROMAHOLD_FILTER)             += vf_chromakey.o
>  OBJS-$(CONFIG_CHROMAKEY_FILTER)              += vf_chromakey.o
> diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> index d7db46c..b38550b 100644
> --- a/libavfilter/allfilters.c
> +++ b/libavfilter/allfilters.c
> @@ -196,6 +196,7 @@ extern const AVFilter ff_vf_boxblur;
>  extern const AVFilter ff_vf_boxblur_opencl;
>  extern const AVFilter ff_vf_bwdif;
>  extern const AVFilter ff_vf_cas;
> +extern const AVFilter ff_vf_ccrepack;
>  extern const AVFilter ff_vf_chromaber_vulkan;
>  extern const AVFilter ff_vf_chromahold;
>  extern const AVFilter ff_vf_chromakey;
> diff --git a/libavfilter/vf_ccrepack.c b/libavfilter/vf_ccrepack.c
> new file mode 100644
> index 0000000..fb9120c
> --- /dev/null
> +++ b/libavfilter/vf_ccrepack.c
> @@ -0,0 +1,100 @@
> +/*
> + * CEA-708 Closed Caption Repacker
> + * Copyright (c) 2023 LTN Global Communications
> + *
> + * Author: Devin Heitmueller <dheitmueller@ltnglobal.com>
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> + */
> +
> +/*
> + * Repackage CEA-708 arrays, which deals with incorrect cc_count for a
> given
> + * output framerate, and incorrect 708 padding.
> + *
> + * See CEA CEA-10-A "EIA-708-B Implementation Guidance", Section 26.5
> + * "Grouping DTVCC Data Within user_data() Structure"
> + */
> +
> +#include "avfilter.h"
> +#include "internal.h"
> +#include "ccfifo.h"
> +#include "libavutil/opt.h"
> +
> +typedef struct CCRepackContext
> +{
> +    const AVClass *class;
> +    AVCCFifo *cc_fifo;
> +} CCRepackContext;
> +
> +static const AVOption ccrepack_options[] = {
> +    {  NULL }
> +};
> +
>
unused code as no option


> +AVFILTER_DEFINE_CLASS(ccrepack);
> +
> +static int config_input(AVFilterLink *link)
> +{
> +    CCRepackContext *ctx = link->dst->priv;
> +
> +    if (!(ctx->cc_fifo = ff_ccfifo_alloc(&link->frame_rate, ctx)))
> +        av_log(ctx, AV_LOG_VERBOSE, "Failure to setup CC FIFO queue\n");
> +
>

it's better to return error and print error log here.


> +    return 0;
> +}
> +
> +static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
> +{
> +    CCRepackContext *ctx = inlink->dst->priv;
> +    AVFilterLink *outlink = inlink->dst->outputs[0];
> +
> +    ff_ccfifo_extract(ctx->cc_fifo, frame);
> +    ff_ccfifo_inject(ctx->cc_fifo, frame);
> +
>
It's better to check return of function here.


> +    return ff_filter_frame(outlink, frame);
> +}
> +
> +static av_cold void uninit(AVFilterContext *ctx)
> +{
> +    CCRepackContext *s = ctx->priv;
> +    ff_ccfifo_freep(&s->cc_fifo);
> +}
> +
> +static const AVFilterPad avfilter_vf_ccrepack_inputs[] = {
> +    {
> +        .name         = "default",
> +        .type         = AVMEDIA_TYPE_VIDEO,
> +        .filter_frame = filter_frame,
> +        .config_props = config_input,
> +    },
> +};
> +
> +static const AVFilterPad avfilter_vf_ccrepack_outputs[] = {
> +    {
> +        .name = "default",
> +        .type = AVMEDIA_TYPE_VIDEO,
> +    },
> +};
> +
> +AVFilter ff_vf_ccrepack = {
> +    .name        = "ccrepack",
> +    .description = NULL_IF_CONFIG_SMALL("Repack CEA-708 closed caption
> metadata"),
> +    .uninit      = uninit,
> +    .priv_size   = sizeof(CCRepackContext),
> +    .priv_class  = &ccrepack_class,
> +    FILTER_INPUTS(avfilter_vf_ccrepack_inputs),
> +    FILTER_OUTPUTS(avfilter_vf_ccrepack_outputs),
> +};
> --
> 1.8.3.1
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

Hi Lance,

On Sun, Apr 30, 2023 at 6:42 PM Lance Wang <lance.lmwang@gmail.com> wrote:
>
> On Fri, Apr 28, 2023 at 11:43 PM Devin Heitmueller <
> devin.heitmueller@ltnglobal.com> wrote:
>
> > THis filter can correct certain issues seen from upstream sources
> > where the cc_count is not properly set or the CEA-608 tuples are
> > not at the start of the payload as expected.
> >
> > Make use of the ccfifo to extract and immediately repack the CEA-708
> > side data, thereby removing any extra padding and ensuring the 608
> > tuples are at the front of the payload.
> >
> > Signed-off-by: Devin Heitmueller <dheitmueller@ltnglobal.com>
> > ---
> >  doc/filters.texi          |  10 +++++
> >  libavfilter/Makefile      |   1 +
> >  libavfilter/allfilters.c  |   1 +
> >  libavfilter/vf_ccrepack.c | 100
> > ++++++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 112 insertions(+)
> >  create mode 100644 libavfilter/vf_ccrepack.c
> >
> > diff --git a/doc/filters.texi b/doc/filters.texi
> > index 5dde799..9f67a00 100644
> > --- a/doc/filters.texi
> > +++ b/doc/filters.texi
> > @@ -8936,6 +8936,16 @@ Only deinterlace frames marked as interlaced.
> >  The default value is @code{all}.
> >  @end table
> >
> > +@section ccrepack
> > +
> > +Repack CEA-708 closed captioning side data
> > +
> > +This filter fixes various issues seen with commerical encoders
> > +related to upstream malformed CEA-708 payloads, specifically
> > +incorrect number of tuples (wrong cc_count for the target FPS),
> > +and incorrect ordering of tuples (i.e. the CEA-608 tuples are not at
> > +the first entries in the payload).
> > +
> >  @section cas
> >
> >  Apply Contrast Adaptive Sharpen filter to video stream.
> > diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> > index 628ade8..3d0b3ad 100644
> > --- a/libavfilter/Makefile
> > +++ b/libavfilter/Makefile
> > @@ -212,6 +212,7 @@ OBJS-$(CONFIG_BOXBLUR_OPENCL_FILTER)         +=
> > vf_avgblur_opencl.o opencl.o \
> >                                                  opencl/avgblur.o boxblur.o
> >  OBJS-$(CONFIG_BWDIF_FILTER)                  += vf_bwdif.o yadif_common.o
> >  OBJS-$(CONFIG_CAS_FILTER)                    += vf_cas.o
> > +OBJS-$(CONFIG_CCREPACK_FILTER)               += vf_ccrepack.o
> >  OBJS-$(CONFIG_CHROMABER_VULKAN_FILTER)       += vf_chromaber_vulkan.o
> > vulkan.o vulkan_filter.o
> >  OBJS-$(CONFIG_CHROMAHOLD_FILTER)             += vf_chromakey.o
> >  OBJS-$(CONFIG_CHROMAKEY_FILTER)              += vf_chromakey.o
> > diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> > index d7db46c..b38550b 100644
> > --- a/libavfilter/allfilters.c
> > +++ b/libavfilter/allfilters.c
> > @@ -196,6 +196,7 @@ extern const AVFilter ff_vf_boxblur;
> >  extern const AVFilter ff_vf_boxblur_opencl;
> >  extern const AVFilter ff_vf_bwdif;
> >  extern const AVFilter ff_vf_cas;
> > +extern const AVFilter ff_vf_ccrepack;
> >  extern const AVFilter ff_vf_chromaber_vulkan;
> >  extern const AVFilter ff_vf_chromahold;
> >  extern const AVFilter ff_vf_chromakey;
> > diff --git a/libavfilter/vf_ccrepack.c b/libavfilter/vf_ccrepack.c
> > new file mode 100644
> > index 0000000..fb9120c
> > --- /dev/null
> > +++ b/libavfilter/vf_ccrepack.c
> > @@ -0,0 +1,100 @@
> > +/*
> > + * CEA-708 Closed Caption Repacker
> > + * Copyright (c) 2023 LTN Global Communications
> > + *
> > + * Author: Devin Heitmueller <dheitmueller@ltnglobal.com>
> > + *
> > + * This file is part of FFmpeg.
> > + *
> > + * FFmpeg is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU Lesser General Public
> > + * License as published by the Free Software Foundation; either
> > + * version 2.1 of the License, or (at your option) any later version.
> > + *
> > + * FFmpeg is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> > + * Lesser General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU Lesser General Public
> > + * License along with FFmpeg; if not, write to the Free Software
> > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> > 02110-1301 USA
> > + */
> > +
> > +/*
> > + * Repackage CEA-708 arrays, which deals with incorrect cc_count for a
> > given
> > + * output framerate, and incorrect 708 padding.
> > + *
> > + * See CEA CEA-10-A "EIA-708-B Implementation Guidance", Section 26.5
> > + * "Grouping DTVCC Data Within user_data() Structure"
> > + */
> > +
> > +#include "avfilter.h"
> > +#include "internal.h"
> > +#include "ccfifo.h"
> > +#include "libavutil/opt.h"
> > +
> > +typedef struct CCRepackContext
> > +{
> > +    const AVClass *class;
> > +    AVCCFifo *cc_fifo;
> > +} CCRepackContext;
> > +
> > +static const AVOption ccrepack_options[] = {
> > +    {  NULL }
> > +};
> > +
> >
> unused code as no option
>
>
> > +AVFILTER_DEFINE_CLASS(ccrepack);
> > +
> > +static int config_input(AVFilterLink *link)
> > +{
> > +    CCRepackContext *ctx = link->dst->priv;
> > +
> > +    if (!(ctx->cc_fifo = ff_ccfifo_alloc(&link->frame_rate, ctx)))
> > +        av_log(ctx, AV_LOG_VERBOSE, "Failure to setup CC FIFO queue\n");
> > +
> >
>
> it's better to return error and print error log here.
>
>
> > +    return 0;
> > +}
> > +
> > +static int filter_frame(AVFilterLink *inlink, AVFrame *frame)
> > +{
> > +    CCRepackContext *ctx = inlink->dst->priv;
> > +    AVFilterLink *outlink = inlink->dst->outputs[0];
> > +
> > +    ff_ccfifo_extract(ctx->cc_fifo, frame);
> > +    ff_ccfifo_inject(ctx->cc_fifo, frame);
> > +
> >
> It's better to check return of function here.
>
>
> > +    return ff_filter_frame(outlink, frame);
> > +}
> > +
> > +static av_cold void uninit(AVFilterContext *ctx)
> > +{
> > +    CCRepackContext *s = ctx->priv;
> > +    ff_ccfifo_freep(&s->cc_fifo);
> > +}
> > +
> > +static const AVFilterPad avfilter_vf_ccrepack_inputs[] = {
> > +    {
> > +        .name         = "default",
> > +        .type         = AVMEDIA_TYPE_VIDEO,
> > +        .filter_frame = filter_frame,
> > +        .config_props = config_input,
> > +    },
> > +};
> > +
> > +static const AVFilterPad avfilter_vf_ccrepack_outputs[] = {
> > +    {
> > +        .name = "default",
> > +        .type = AVMEDIA_TYPE_VIDEO,
> > +    },
> > +};
> > +
> > +AVFilter ff_vf_ccrepack = {
> > +    .name        = "ccrepack",
> > +    .description = NULL_IF_CONFIG_SMALL("Repack CEA-708 closed caption
> > metadata"),
> > +    .uninit      = uninit,
> > +    .priv_size   = sizeof(CCRepackContext),
> > +    .priv_class  = &ccrepack_class,
> > +    FILTER_INPUTS(avfilter_vf_ccrepack_inputs),
> > +    FILTER_OUTPUTS(avfilter_vf_ccrepack_outputs),
> > +};
> > --
> > 1.8.3.1
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

These are all reasonable comments, and in fact largely the result of
my not reflecting changes I made to the API in the last round, where
we now have the ff_ccfifo_alloc() call succeed even if the output
framerate is not supported.

I'll roll these fixes into the next series.

Devin