[FFmpeg-devel,v4,4/5] lavc: Implement Dolby Vision RPU parsing

From: Niklas Haas <git@haasn.dev>

Based on a mixture of guesswork, partial documentation in patents, and
reverse engineering of real-world samples. Confirmed working for all the
samples I've thrown at it.

Contains some annoying machinery to persist these values in between
frames, which is needed in theory even though I've never actually seen a
sample that relies on it in practice. May or may not work.

Since the distinction matters greatly for parsing the color matrix
values, this includes a small helper function to guess the right profile
from the RPU itself in case the user has forgotten to forward the dovi
configuration record to the decoder. (Which in practice, only ffmpeg.c
and ffplay do..)

Notable omissions:
- CRC32 verificaiton. This is based on the MPEG2 CRC32 type, which does
  not seem to be implemented in lavc. (And I don't care enough to do so)
- Linear interpolation support. Nothing documents this (beyond its
  existence) and no samples use it, so impossible to implement.
- All of the extension metadata blocks, but these contain values that
  are basically congruent with ST2094, HDR10, or other existing forms of
  side data, so I will defer parsing/attaching them to a future commit.
- The patent describes a mechanism for predicting coefficients from
  previous RPUs, but the bit for the flag whether to use the
  prediction deltas or signal entirely new coefficients does not seem to
  be present in actual RPUs, so we ignore this subsystem entirely.

Heavily influenced by https://github.com/quietvoid/dovi_tool
Documentation drawn from US Patent 10,701,399 B2 and ETSI GS CCM 001

Signed-off-by: Niklas Haas <git@haasn.dev>
---
 libavcodec/Makefile |   1 +
 libavcodec/dovi.c   | 415 ++++++++++++++++++++++++++++++++++++++++++++
 libavcodec/dovi.h   |  68 ++++++++
 3 files changed, 484 insertions(+)
 create mode 100644 libavcodec/dovi.c
 create mode 100644 libavcodec/dovi.h

Niklas Haas:
> From: Niklas Haas <git@haasn.dev>
> 
> Based on a mixture of guesswork, partial documentation in patents, and
> reverse engineering of real-world samples. Confirmed working for all the
> samples I've thrown at it.
> 
> Contains some annoying machinery to persist these values in between
> frames, which is needed in theory even though I've never actually seen a
> sample that relies on it in practice. May or may not work.
> 
> Since the distinction matters greatly for parsing the color matrix
> values, this includes a small helper function to guess the right profile
> from the RPU itself in case the user has forgotten to forward the dovi
> configuration record to the decoder. (Which in practice, only ffmpeg.c
> and ffplay do..)
> 
> Notable omissions:
> - CRC32 verificaiton. This is based on the MPEG2 CRC32 type, which does

verification

>   not seem to be implemented in lavc. (And I don't care enough to do so)

lavu

> - Linear interpolation support. Nothing documents this (beyond its
>   existence) and no samples use it, so impossible to implement.
> - All of the extension metadata blocks, but these contain values that
>   are basically congruent with ST2094, HDR10, or other existing forms of
>   side data, so I will defer parsing/attaching them to a future commit.
> - The patent describes a mechanism for predicting coefficients from
>   previous RPUs, but the bit for the flag whether to use the
>   prediction deltas or signal entirely new coefficients does not seem to
>   be present in actual RPUs, so we ignore this subsystem entirely.
> 
> Heavily influenced by https://github.com/quietvoid/dovi_tool
> Documentation drawn from US Patent 10,701,399 B2 and ETSI GS CCM 001
> 
> Signed-off-by: Niklas Haas <git@haasn.dev>
> ---
>  libavcodec/Makefile |   1 +
>  libavcodec/dovi.c   | 415 ++++++++++++++++++++++++++++++++++++++++++++
>  libavcodec/dovi.h   |  68 ++++++++
>  3 files changed, 484 insertions(+)
>  create mode 100644 libavcodec/dovi.c
>  create mode 100644 libavcodec/dovi.h
> 
> diff --git a/libavcodec/Makefile b/libavcodec/Makefile
> index 4122a9b144..c327e0d764 100644
> --- a/libavcodec/Makefile
> +++ b/libavcodec/Makefile
> @@ -40,6 +40,7 @@ OBJS = ac3_parser.o                                                     \
>         d3d11va.o                                                        \
>         decode.o                                                         \
>         dirac.o                                                          \
> +       dovi.o                                                           \

Why is this always compiled when only the HEVC decoder needs it?
(The way it is now will cause linking errors when CONFIG_GOLOMB is false.)

>         dv_profile.o                                                     \
>         encode.o                                                         \
>         imgconvert.o                                                     \
> diff --git a/libavcodec/dovi.c b/libavcodec/dovi.c
> new file mode 100644
> index 0000000000..cebe42d735
> --- /dev/null
> +++ b/libavcodec/dovi.c
> @@ -0,0 +1,415 @@
> +/*
> + * Dolby Vision RPU decoder
> + *
> + * Copyright (C) 2021 Jan Ekström
> + * Copyright (C) 2021 Niklas Haas
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include "dovi.h"
> +#include "golomb.h"
> +#include "get_bits.h"
> +
> +enum {
> +    RPU_COEFF_FIXED = 0,
> +    RPU_COEFF_FLOAT = 1,
> +};
> +
> +/**
> + * Private contents of vdr_ref.
> + */
> +typedef struct DOVIVdrRef {
> +    AVDOVIDataMapping mapping;
> +    AVDOVIColorMetadata color;
> +} DOVIVdrRef;
> +
> +void ff_dovi_ctx_unref(DOVIContext *s)
> +{
> +    for (int i = 0; i < DOVI_MAX_DM_ID; i++)
> +        av_buffer_unref(&s->vdr_ref[i]);
> +
> +    /* Preserve the AVCodecContext explicitly, reset everything else */
> +    *s = (DOVIContext) {
> +        .avctx = s->avctx,
> +    };
> +}
> +
> +int ff_dovi_ctx_replace(DOVIContext *s, const DOVIContext *s0)
> +{
> +    int ret;
> +    s->config = s0->config;
> +    for (int i = 0; i < DOVI_MAX_DM_ID; i++) {
> +        if ((ret = av_buffer_replace(&s->vdr_ref[i], s0->vdr_ref[i])) < 0)
> +            goto fail;
> +    }
> +
> +    s->mapping = s0->mapping;
> +    s->color = s0->color;
> +    return 0;
> +
> +fail:
> +    s->mapping = NULL; /* sanity */
> +    s->color = NULL;
> +    return ret;
> +}
> +
> +int ff_dovi_attach_side_data(DOVIContext *s, AVFrame *frame)
> +{
> +    AVFrameSideData *sd;
> +    AVDOVIMetadata *dovi;
> +    if (!s->mapping || !s->color)
> +        return 0; /* incomplete dovi metadata */
> +
> +    sd = av_frame_new_side_data(frame, AV_FRAME_DATA_DOVI_METADATA, sizeof(*dovi));

Just to be sure: sizeof(AVDOVIMetadata) is supposed to be part of the ABI?

> +    if (!sd)
> +        return AVERROR(ENOMEM);
> +
> +    dovi = (AVDOVIMetadata *) sd->data;
> +    memcpy(&dovi->header, &s->header, sizeof(dovi->header));
> +    memcpy(&dovi->mapping, s->mapping, sizeof(dovi->mapping));
> +    memcpy(&dovi->color, s->color, sizeof(dovi->color));
> +    return 0;
> +}
> +
> +int ff_dovi_rpu_parse(DOVIContext *s, const uint8_t *rpu, size_t rpu_size)
> +{
> +    AVDOVIRpuDataHeader *hdr = &s->header;
> +    GetBitContext *gb = &(GetBitContext){0};
> +    DOVIVdrRef *vdr;
> +    int ret;
> +
> +    uint8_t nal_prefix;
> +    uint8_t rpu_type;
> +    uint8_t vdr_seq_info_present;
> +    uint8_t vdr_dm_metadata_present;
> +    uint8_t use_prev_vdr_rpu;
> +    uint8_t use_nlq;
> +    uint8_t profile;
> +    if ((ret = init_get_bits8(gb, rpu, rpu_size)) < 0)
> +        return ret;
> +
> +    /* RPU header, common values */
> +    nal_prefix = get_bits(gb, 8);
> +    VALIDATE(nal_prefix, 25, 25);
> +    rpu_type = get_bits(gb, 6);
> +    if (rpu_type != 2) {
> +        av_log(s->avctx, AV_LOG_WARNING, "Unrecognized RPU type "
> +               "%"PRIu8", ignoring\n", rpu_type);
> +        return 0;
> +    }
> +
> +    hdr->rpu_type = rpu_type;
> +    hdr->rpu_format = get_bits(gb, 11);
> +
> +    /* Values specific to RPU type 2 */
> +    hdr->vdr_rpu_profile = get_bits(gb, 4);
> +    hdr->vdr_rpu_level = get_bits(gb, 4);
> +
> +    vdr_seq_info_present = get_bits1(gb);
> +    if (vdr_seq_info_present) {
> +        hdr->chroma_resampling_explicit_filter_flag = get_bits1(gb);
> +        hdr->coef_data_type = get_bits(gb, 2);
> +        VALIDATE(hdr->coef_data_type, RPU_COEFF_FIXED, RPU_COEFF_FLOAT);
> +        switch (hdr->coef_data_type) {
> +        case RPU_COEFF_FIXED:
> +            hdr->coef_log2_denom = get_ue_golomb(gb);
> +            VALIDATE(hdr->coef_log2_denom, 13, 32);
> +            break;
> +        case RPU_COEFF_FLOAT:
> +            hdr->coef_log2_denom = 32; /* arbitrary, choose maximum precision */
> +            break;
> +        }
> +
> +        hdr->vdr_rpu_normalized_idc = get_bits(gb, 2);
> +        hdr->bl_video_full_range_flag = get_bits1(gb);
> +
> +        if ((hdr->rpu_format & 0x700) == 0) {
> +            int bl_bit_depth_minus8 = get_ue_golomb_31(gb);
> +            int el_bit_depth_minus8 = get_ue_golomb_31(gb);
> +            int vdr_bit_depth_minus8 = get_ue_golomb_31(gb);
> +            VALIDATE(bl_bit_depth_minus8, 0, 8);
> +            VALIDATE(el_bit_depth_minus8, 0, 8);
> +            VALIDATE(vdr_bit_depth_minus8, 0, 8);
> +            hdr->bl_bit_depth = bl_bit_depth_minus8 + 8;
> +            hdr->el_bit_depth = el_bit_depth_minus8 + 8;
> +            hdr->vdr_bit_depth = vdr_bit_depth_minus8 + 8;
> +            hdr->spatial_resampling_filter_flag = get_bits1(gb);
> +            skip_bits(gb, 3); /* reserved_zero_3bits */
> +            hdr->el_spatial_resampling_filter_flag = get_bits1(gb);
> +            hdr->disable_residual_flag = get_bits1(gb);
> +        }
> +    }
> +
> +    if (!hdr->bl_bit_depth) {
> +        av_log(s->avctx, AV_LOG_ERROR, "Missing RPU VDR sequence info?\n");
> +        goto fail;
> +    }
> +
> +    vdr_dm_metadata_present = get_bits1(gb);
> +    use_prev_vdr_rpu = get_bits1(gb);
> +    use_nlq = (hdr->rpu_format & 0x700) == 0 && !hdr->disable_residual_flag;
> +
> +    profile = s->config.dv_profile ? s->config.dv_profile : guess_profile(hdr);
> +    if (profile == 5 && use_nlq) {
> +        av_log(s->avctx, AV_LOG_ERROR, "Profile 5 RPUs should not use NLQ\n");
> +        goto fail;
> +    }
> +
> +    if (use_prev_vdr_rpu) {
> +        int prev_vdr_rpu_id = get_ue_golomb_31(gb);
> +        VALIDATE(prev_vdr_rpu_id, 0, DOVI_MAX_DM_ID);
> +        if (!s->vdr_ref[prev_vdr_rpu_id]) {
> +            av_log(s->avctx, AV_LOG_ERROR, "Unknown previous RPU ID: %u\n",
> +                   prev_vdr_rpu_id);
> +            goto fail;
> +        }
> +        vdr = (DOVIVdrRef *) s->vdr_ref[prev_vdr_rpu_id]->data;
> +        s->mapping = &vdr->mapping;
> +    } else {
> +        int vdr_rpu_id = get_ue_golomb_31(gb);
> +        VALIDATE(vdr_rpu_id, 0, DOVI_MAX_DM_ID);
> +        if (!s->vdr_ref[vdr_rpu_id]) {
> +            s->vdr_ref[vdr_rpu_id] = av_buffer_allocz(sizeof(DOVIVdrRef));
> +            if (!s->vdr_ref[vdr_rpu_id])
> +                return AVERROR(ENOMEM);
> +        }
> +
> +        vdr = (DOVIVdrRef *) s->vdr_ref[vdr_rpu_id]->data;
> +        s->mapping = &vdr->mapping;
> +        vdr->mapping = (AVDOVIDataMapping) {
> +            .vdr_rpu_id = vdr_rpu_id,
> +            .mapping_color_space = get_ue_golomb_31(gb),
> +            .mapping_chroma_format_idc = get_ue_golomb_31(gb),

"The evaluations of the initialization list expressions are
indeterminately sequenced with respect to one another and thus the order
in which any side effects occur is unspecified." (C11, 6.7.9 23 (older
standards contained equivalent statementes))

> +        };
> +
> +        for (int c = 0; c < 3; c++) {
> +            AVDOVIReshapingCurve *curve = &vdr->mapping.curves[c];
> +            int num_pivots_minus_2 = get_ue_golomb_31(gb);
> +            int pivot = 0;
> +
> +            VALIDATE(num_pivots_minus_2, 0, FF_DOVI_MAX_PIECES - 1);
> +            curve->num_pivots = num_pivots_minus_2 + 2;
> +            for (int i = 0; i < curve->num_pivots; i++) {
> +                pivot += get_bits(gb, hdr->bl_bit_depth);
> +                curve->pivots[i] = av_clip_uint16(pivot);
> +            }
> +        }
> +
> +        if (use_nlq) {
> +            vdr->mapping.nlq_method_idc = get_bits(gb, 3);
> +            /**
> +             * The patent mentions another legal value, NLQ_MU_LAW, but it's
> +             * not documented anywhere how to parse or apply that type of NLQ.
> +             */
> +            VALIDATE(vdr->mapping.nlq_method_idc, 0, AV_DOVI_NLQ_LINEAR_DZ);
> +        } else {
> +            vdr->mapping.nlq_method_idc = AV_DOVI_NLQ_NONE;
> +        }
> +
> +        vdr->mapping.num_x_partitions = get_ue_golomb_long(gb) + 1;
> +        vdr->mapping.num_y_partitions = get_ue_golomb_long(gb) + 1;
> +        /* End of rpu_data_header(), start of vdr_rpu_data_payload() */
> +
> +        for (int c = 0; c < 3; c++) {
> +            AVDOVIReshapingCurve *curve = &vdr->mapping.curves[c];
> +            for (int i = 0; i < curve->num_pivots - 1; i++) {
> +                int mapping_idc = get_ue_golomb_31(gb);
> +                VALIDATE(mapping_idc, 0, 1);
> +                curve->mapping_idc[i] = mapping_idc;
> +                switch (mapping_idc) {
> +                case AV_DOVI_MAPPING_POLYNOMIAL: {
> +                    int poly_order_minus1 = get_ue_golomb_31(gb);
> +                    VALIDATE(poly_order_minus1, 0, 1);
> +                    curve->poly_order[i] = poly_order_minus1 + 1;
> +                    if (poly_order_minus1 == 0) {
> +                        int linear_interp_flag = get_bits1(gb);
> +                        if (linear_interp_flag) {
> +                            /* lack of documentation/samples */
> +                            av_log(s->avctx, AV_LOG_WARNING, "Linear "
> +                                   "interpolation is unimplemented, please "
> +                                   "open a bug report with the sample.\n");

avpriv_request_sample()

> +                            ff_dovi_ctx_unref(s);
> +                            return AVERROR_PATCHWELCOME;
> +                        }
> +                    }
> +                    for (int k = 0; k <= curve->poly_order[i]; k++)
> +                        curve->poly_coef[i][k] = get_se_coef(gb, hdr);
> +                    break;
> +                }
> +                case AV_DOVI_MAPPING_MMR: {
> +                    int mmr_order_minus1 = get_bits(gb, 2);
> +                    VALIDATE(mmr_order_minus1, 0, 2);
> +                    curve->mmr_order[i] = mmr_order_minus1 + 1;
> +                    curve->mmr_constant[i] = get_se_coef(gb, hdr);
> +                    for (int j = 0; j < curve->mmr_order[i]; j++) {
> +                        for (int k = 0; k < 7; k++)
> +                            curve->mmr_coef[i][j][k] = get_se_coef(gb, hdr);
> +                    }
> +                    break;
> +                }
> +                }
> +            }
> +        }
> +
> +        if (use_nlq) {
> +            for (int c = 0; c < 3; c++) {
> +                AVDOVINLQParams *nlq = &vdr->mapping.nlq[c];
> +                nlq->nlq_offset = get_bits(gb, hdr->el_bit_depth);
> +                nlq->vdr_in_max = get_ue_coef(gb, hdr);
> +                switch (vdr->mapping.nlq_method_idc) {
> +                case AV_DOVI_NLQ_LINEAR_DZ:
> +                    nlq->linear_deadzone_slope = get_ue_coef(gb, hdr);
> +                    nlq->linear_deadzone_threshold = get_ue_coef(gb, hdr);
> +                    break;
> +                }
> +            }
> +        }
> +    }
> +
> +    if (vdr_dm_metadata_present) {
> +        AVDOVIColorMetadata *color;
> +        int affected_dm_id = get_ue_golomb_31(gb);
> +        int current_dm_id = get_ue_golomb_31(gb);
> +        VALIDATE(affected_dm_id, 0, DOVI_MAX_DM_ID);
> +        VALIDATE(current_dm_id, 0, DOVI_MAX_DM_ID);
> +        if (!s->vdr_ref[affected_dm_id]) {
> +            s->vdr_ref[affected_dm_id] = av_buffer_allocz(sizeof(DOVIVdrRef));
> +            if (!s->vdr_ref[affected_dm_id])
> +                return AVERROR(ENOMEM);

Does the context not need to be reset upon allocation error?

> +        }
> +
> +        if (!s->vdr_ref[current_dm_id]) {
> +            av_log(s->avctx, AV_LOG_ERROR, "Unknown previous RPU DM ID: %u\n",
> +                   current_dm_id);
> +            goto fail;
> +        }
> +
> +        /* Update current pointer based on current_dm_id */
> +        vdr = (DOVIVdrRef *) s->vdr_ref[current_dm_id]->data;
> +        s->color = &vdr->color;
> +
> +        /* Update values of affected_dm_id */
> +        vdr = (DOVIVdrRef *) s->vdr_ref[affected_dm_id]->data;
> +        color = &vdr->color;
> +        color->dm_metadata_id = affected_dm_id;
> +        color->scene_refresh_flag = get_ue_golomb_31(gb);
> +        for (int i = 0; i < 9; i++)
> +            color->ycc_to_rgb_matrix[i] = av_make_q(get_sbits(gb, 16), 1 << 13);
> +        for (int i = 0; i < 3; i++) {
> +            int denom = profile == 4 ? (1 << 30) : (1 << 28);
> +            unsigned offset = get_bits_long(gb, 32);
> +            while (offset > INT_MAX) {

if (offset > INT_MAX)

> +                /* Ensure the result fits inside AVRational */
> +                offset >>= 1;
> +                denom >>= 1;
> +            }
> +            color->ycc_to_rgb_offset[i] = av_make_q(offset, denom);
> +        }
> +        for (int i = 0; i < 9; i++)
> +            color->rgb_to_lms_matrix[i] = av_make_q(get_sbits(gb, 16), 1 << 14);
> +
> +        color->signal_eotf = get_bits(gb, 16);
> +        color->signal_eotf_param0 = get_bits(gb, 16);
> +        color->signal_eotf_param1 = get_bits(gb, 16);
> +        color->signal_eotf_param2 = get_bits_long(gb, 32);
> +        color->signal_bit_depth = get_bits(gb, 5);
> +        VALIDATE(color->signal_bit_depth, 8, 16);
> +        color->signal_color_space = get_bits(gb, 2);
> +        color->signal_chroma_format = get_bits(gb, 2);
> +        color->signal_full_range_flag = get_bits(gb, 2);
> +        color->source_min_pq = get_bits(gb, 12);
> +        color->source_max_pq = get_bits(gb, 12);
> +        color->source_diagonal = get_bits(gb, 10);
> +    }
> +
> +    /* FIXME: verify CRC32, requires implementation of AV_CRC_32_MPEG_2 */
> +    return 0;
> +
> +fail:
> +    ff_dovi_ctx_unref(s); /* don't leak potentially invalid state */
> +    return AVERROR(EINVAL);
> +}

On Sat, 11 Dec 2021 15:02:14 +0100 Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote:
> verification

Fixed

> lavu

Fixed

> Why is this always compiled when only the HEVC decoder needs it?
> (The way it is now will cause linking errors when CONFIG_GOLOMB is false.)

In principle, parsing the RPU contents is orthogonal from their delivery
within HEVC. According to Dolby the same RPU may be attached to AVC
(though admittedly nobody does so), and VVC also supports it (although
this one is not implemented in lavc yet).

How about instead adding CONFIG_DOLBYVISION as an extra configure option
(similar to CONFIG_GOLOMB), and have hevc_decoder auto-select it?

> Just to be sure: sizeof(AVDOVIMetadata) is supposed to be part of the ABI?

There's actually an argument to be made here both ways. IIRC there are
sometimes unparsed bytes left at the end of the RPU. If somebody figures
out what these are, we might be able to parse them (and add them to the
struct) in the future.

There's also the question of the extension blocks - which currently
don't seem interesting for any client and largely redundant with other
forms of side data.

Maybe it makes sense to have the sizeof this struct not be part of the
ABI.

> "The evaluations of the initialization list expressions are
> indeterminately sequenced with respect to one another and thus the order
> in which any side effects occur is unspecified." (C11, 6.7.9 23 (older
> standards contained equivalent statementes))

Fixed

> avpriv_request_sample()

Fixed

> Does the context not need to be reset upon allocation error?

I think this is subjective. The reason I reset the context for
validation errors is to avoid exposing clients to potentially illegal
values (like num_pivots exceeding the bounds of the static array).

For an allocation error, it's arguably better to keep the previously
decoded (and therefore still valid) state intact? I doubt it matters in
practice because any process which fails to allocate this small amount
of memory is probably not going to get very far anyways.

> if (offset > INT_MAX)

Fixed

Niklas Haas:
> On Sat, 11 Dec 2021 15:02:14 +0100 Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote:
>> verification
> 
> Fixed
> 
>> lavu
> 
> Fixed
> 
>> Why is this always compiled when only the HEVC decoder needs it?
>> (The way it is now will cause linking errors when CONFIG_GOLOMB is false.)
> 
> In principle, parsing the RPU contents is orthogonal from their delivery
> within HEVC. According to Dolby the same RPU may be attached to AVC
> (though admittedly nobody does so), and VVC also supports it (although
> this one is not implemented in lavc yet).
> 
> How about instead adding CONFIG_DOLBYVISION as an extra configure option
> (similar to CONFIG_GOLOMB), and have hevc_decoder auto-select it?
> 

Sounds fine to me (I thought about just adding this file to the list of
files compiled when the HEVC decoder is selected, but the end result is
the same).

- Andreas