From patchwork Mon Jan 25 14:15:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nuo Mi X-Patchwork-Id: 25167 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 2CFB044BD19 for ; Mon, 25 Jan 2021 16:17:32 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 153B068A345; Mon, 25 Jan 2021 16:17:32 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 4FA3368A31F for ; Mon, 25 Jan 2021 16:17:30 +0200 (EET) Received: by mail-pl1-f179.google.com with SMTP id b8so7649698plh.12 for ; Mon, 25 Jan 2021 06:17:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UvyYvJMYVrRVMQrMXIG/b01WbXOEblLNA53FXPTzNTg=; b=PM1pthJEGS1PFTDkNFm0ey/NQ3rHL3vBCas1Fa9y6SgDxw1ps9Lp5ZOdOspIAKtaGM BQbGX2blfGUbdYoiP8BLG59Qk6VgodGm4XUj6BypoMhDnAvEkGs54A5PJKkfBpcJQX3B Z82OwUk0nmtjvEAiLjI3+rxMz8CLVbnZlZVzVHITjoZsXACLQXB18/PdG6I1CBEXaaOf jb8wiKcBcBSwfrFT+P9NMHMagdzAQub/eg4jo+kerzVzUGhPj9CdyH6Z048/6GovI7Io TaPk1ze8P72hPAYx9qz6z63Woa+qDX5wIOh4wYP94mHP5p25ACe3+j+qnxu6IScpefY/ gtvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UvyYvJMYVrRVMQrMXIG/b01WbXOEblLNA53FXPTzNTg=; b=bEqUUMbKTxYA8IlXj7yPEghngf7g7BGIQlu8fYpyJIe901x1IzLRqgDi6xHaWk/78M pR3MliABzS5dT2UBfZoVhhZsNOgSww26P9DhTB1nWcV8FHFZWdfoexHKqwlLUh4b19ek Vei2Qz3gfHE0fyebAV+URwgnf2sTTemTW/p1hsrAMJzj16uG1q0U51CEUbBR8ECwCE8w Vw67ME8S4sk/4MDQlk3U/FRnIGGzFiaE1X22O9T1462D2B9ON0Hh2kXneazl/m0mW8/O UcoOiRrO9Y58kGcsxGJRX0uLzIFmq9OBwKUn49yAWm6hiYejOVITOmqPrpMSNYJe/on/ wXHw== X-Gm-Message-State: AOAM533T2UoY6OzWHo40X9YTw1II46kwwiF655HlZhRq8J+6TFKxq7E+ a6Tiqj6Ddf+PbBfAtNHrFX/2JDUIU44= X-Google-Smtp-Source: ABdhPJzqED0XJbfuBFk1ch6E2CqtC/FONAWBhrf7G4vNOaPZ6ZcUCHYzZT2uWICcwIr+mAlVDywbFw== X-Received: by 2002:a17:90a:fd0b:: with SMTP id cv11mr422334pjb.26.1611584248182; Mon, 25 Jan 2021 06:17:28 -0800 (PST) Received: from smith.tendawifi.com (23.83.245.51.16clouds.com. [23.83.245.51]) by smtp.gmail.com with ESMTPSA id z6sm5523328pfr.133.2021.01.25.06.17.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 06:17:27 -0800 (PST) From: Nuo Mi To: ffmpeg-devel@ffmpeg.org Date: Mon, 25 Jan 2021 22:15:07 +0800 Message-Id: <20210125141507.11012-10-nuomi2021@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125141507.11012-1-nuomi2021@gmail.com> References: <20210111152357.3965-1-nuomi2021@gmail.com> <20210125141507.11012-1-nuomi2021@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v4 9/9] avcodec: add vvc metadata bsf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nuo Mi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" use following command to test: ffmpeg -i in.bin -c:v copy -bsf vvc_metadata -f vvc out.bin 94.51%(293/310) can bit match with original clips 5.48%(17/310) are not bit match, the original clips has redundant emulation prevent bytes --- configure | 1 + libavcodec/Makefile | 1 + libavcodec/bitstream_filters.c | 1 + libavcodec/h266_metadata_bsf.c | 227 +++++++++++++++++++++++++++++++++ 4 files changed, 230 insertions(+) create mode 100644 libavcodec/h266_metadata_bsf.c diff --git a/configure b/configure index 74ff58d7b7..cea5a00119 100755 --- a/configure +++ b/configure @@ -3184,6 +3184,7 @@ mjpeg2jpeg_bsf_select="jpegtables" mpeg2_metadata_bsf_select="cbs_mpeg2" trace_headers_bsf_select="cbs" vp9_metadata_bsf_select="cbs_vp9" +vvc_metadata_bsf_select="cbs_h266" # external libraries aac_at_decoder_deps="audiotoolbox" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index e736c8300a..1b90d61a26 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1169,6 +1169,7 @@ OBJS-$(CONFIG_VP9_METADATA_BSF) += vp9_metadata_bsf.o OBJS-$(CONFIG_VP9_RAW_REORDER_BSF) += vp9_raw_reorder_bsf.o OBJS-$(CONFIG_VP9_SUPERFRAME_BSF) += vp9_superframe_bsf.o OBJS-$(CONFIG_VP9_SUPERFRAME_SPLIT_BSF) += vp9_superframe_split_bsf.o +OBJS-$(CONFIG_VVC_METADATA_BSF) += h266_metadata_bsf.o # thread libraries OBJS-$(HAVE_LIBC_MSVCRT) += file_open.o diff --git a/libavcodec/bitstream_filters.c b/libavcodec/bitstream_filters.c index b26d6a910e..001a7bb3a4 100644 --- a/libavcodec/bitstream_filters.c +++ b/libavcodec/bitstream_filters.c @@ -60,6 +60,7 @@ extern const AVBitStreamFilter ff_vp9_metadata_bsf; extern const AVBitStreamFilter ff_vp9_raw_reorder_bsf; extern const AVBitStreamFilter ff_vp9_superframe_bsf; extern const AVBitStreamFilter ff_vp9_superframe_split_bsf; +extern const AVBitStreamFilter ff_vvc_metadata_bsf; #include "libavcodec/bsf_list.c" diff --git a/libavcodec/h266_metadata_bsf.c b/libavcodec/h266_metadata_bsf.c new file mode 100644 index 0000000000..1a93aa7e26 --- /dev/null +++ b/libavcodec/h266_metadata_bsf.c @@ -0,0 +1,227 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" +#include "libavutil/opt.h" + +#include "bsf.h" +#include "bsf_internal.h" +#include "cbs.h" +#include "cbs_bsf.h" +#include "cbs_h266.h" +#include "vvc.h" + +#define IS_SLICE(nut) (nut <= VVC_RASL_NUT || (nut >= VVC_IDR_W_RADL && nut <= VVC_GDR_NUT)) +#define IS_IDR(nut) (nut == VVC_IDR_W_RADL || nut == VVC_IDR_N_LP) + +typedef struct VVCMetadataContext { + CBSBSFContext common; + + H266RawAUD aud_nal; + + int aud; + + uint8_t prev_layer_id; + int prev_tid0_poc; + int prev_poc; +} VVCMetadataContext; + +//8.3.1 Decoding process for picture order count. +//VTM did not follow the spec, and it's much simpler than spec. +//We follow the VTM. +static int get_slice_poc(AVBSFContext *bsf, const H266RawPH *ph, + const H266RawSliceHeader *slice, int *poc) +{ + VVCMetadataContext *ctx = bsf->priv_data; + CodedBitstreamH266Context *h266 = ctx->common.input->priv_data; + int poc_msb, max_poc_lsb, poc_lsb; + const H266RawSPS* sps = h266->active_sps; + + if (!sps) { + av_log(bsf, AV_LOG_ERROR, "no pps for pu.\n"); + return AVERROR_INVALIDDATA; + } + + max_poc_lsb = 1 << (sps->sps_log2_max_pic_order_cnt_lsb_minus4 + 4); + poc_lsb = ph->ph_pic_order_cnt_lsb; + if (IS_IDR(slice->nal_unit_header.nal_unit_type)) { + if (ph->ph_poc_msb_cycle_present_flag) + poc_msb = ph->ph_poc_msb_cycle_val * max_poc_lsb; + else + poc_msb = 0; + } else { + int prev_poc = ctx->prev_tid0_poc; + int prev_poc_lsb = prev_poc & (max_poc_lsb - 1); + int prev_poc_msb = prev_poc - prev_poc_lsb; + if (ph->ph_poc_msb_cycle_present_flag) { + poc_msb = ph->ph_poc_msb_cycle_val * max_poc_lsb; + } else { + if ((poc_lsb < prev_poc_lsb) && ((prev_poc_lsb - poc_lsb) >= (max_poc_lsb / 2))) + poc_msb = prev_poc_msb + max_poc_lsb; + else if ((poc_lsb > prev_poc_lsb) && ((poc_lsb - prev_poc_lsb) > (max_poc_lsb / 2))) + poc_msb = prev_poc_msb - max_poc_lsb; + else + poc_msb = prev_poc_msb; + } + } + + *poc = poc_msb + poc_lsb; + return 0; +} + +static int is_au_start(AVBSFContext *bsf, CodedBitstreamFragment *pu) +{ + //7.4.2.4.3 + VVCMetadataContext *ctx = bsf->priv_data; + CodedBitstreamH266Context *h266 = ctx->common.input->priv_data; + int i, ret; + const H266RawNALUnitHeader *nal; + const H266RawPH *ph = h266->ph; + const H266RawSlice *slice = NULL; + int poc, nut; + + if (!ph) { + av_log(bsf, AV_LOG_ERROR, "can't find picture header in picutre unit.\n"); + return AVERROR_INVALIDDATA; + } + for (i = 0; i < pu->nb_units; i++) { + nal = pu->units[i].content; + if (!nal) + continue; + if (IS_SLICE(nal->nal_unit_type)) { + slice = pu->units[i].content; + break; + } + } + if (!slice) { + av_log(bsf, AV_LOG_ERROR, "can't find first slice.\n"); + return AVERROR_INVALIDDATA; + } + + if ((ret = get_slice_poc(bsf, ph, &slice->header, &poc)) < 0) + return ret; + + ret = nal->nuh_layer_id <= ctx->prev_layer_id || poc != ctx->prev_poc; + ctx->prev_layer_id = nal->nuh_layer_id; + ctx->prev_poc = poc; + + nut = nal->nal_unit_type; + if (nal->nuh_temporal_id_plus1 == 1 && + !ph->ph_non_ref_pic_flag && nut != VVC_RADL_NUT && nut != VVC_RASL_NUT) { + ctx->prev_tid0_poc = poc; + } + return ret; +} + +static int h266_metadata_update_fragment(AVBSFContext *bsf, AVPacket *pkt, + CodedBitstreamFragment *pu) +{ + VVCMetadataContext *ctx = bsf->priv_data; + int err, i; + + // If an AUD is present, it must be the first NAL unit. + if (pu->units[0].type == VVC_AUD_NUT) { + if (ctx->aud == BSF_ELEMENT_REMOVE) + ff_cbs_delete_unit(pu, 0); + } else { + if (ctx->aud == BSF_ELEMENT_INSERT && is_au_start(bsf, pu)) { + H266RawAUD *aud = &ctx->aud_nal; + int pic_type = 0, temporal_id = 8, layer_id = 0; + CodedBitstreamH266Context *h266 = ctx->common.input->priv_data; + for (i = 0; i < pu->nb_units; i++) { + const H266RawNALUnitHeader *nal = pu->units[i].content; + if (!nal) + continue; + if (nal->nuh_temporal_id_plus1 < temporal_id + 1) + temporal_id = nal->nuh_temporal_id_plus1 - 1; + + if (pu->units[i].type <= VVC_RSV_IRAP_11) { + const H266RawSlice *slice = pu->units[i].content; + layer_id = nal->nuh_layer_id; + if (slice->header.sh_slice_type == VVC_SLICE_TYPE_B && + pic_type < 2) + pic_type = 2; + if (slice->header.sh_slice_type == VVC_SLICE_TYPE_P && + pic_type < 1) + pic_type = 1; + } + } + + aud->nal_unit_header = (H266RawNALUnitHeader) { + .nal_unit_type = VVC_AUD_NUT, + .nuh_layer_id = layer_id, + .nuh_temporal_id_plus1 = temporal_id + 1, + }; + aud->aud_pic_type = pic_type; + aud->aud_irap_or_gdr_flag = h266->ph->ph_gdr_or_irap_pic_flag; + + err = ff_cbs_insert_unit_content(pu, 0, VVC_AUD_NUT, aud, NULL); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to insert AUD.\n"); + return err; + } + } + } + return 0; +} + +static const CBSBSFType h266_metadata_type = { + .codec_id = AV_CODEC_ID_VVC, + .fragment_name = "picture unit", + .unit_name = "NAL unit", + .update_fragment = &h266_metadata_update_fragment, +}; + +static int vvc_metadata_init(AVBSFContext *bsf) +{ + VVCMetadataContext *ctx = bsf->priv_data; + ctx->prev_layer_id = UINT8_MAX; + ctx->prev_poc = INT_MAX; + ctx->prev_tid0_poc = INT_MAX; + return ff_cbs_bsf_generic_init(bsf, &h266_metadata_type); +} + +#define OFFSET(x) offsetof(VVCMetadataContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_BSF_PARAM) +static const AVOption vvc_metadata_options[] = { + BSF_ELEMENT_OPTIONS_PIR("aud", "Access Unit Delimiter NAL units", + aud, FLAGS), + + { NULL } +}; + +static const AVClass vvc_metadata_class = { + .class_name = "vvc_metadata_bsf", + .item_name = av_default_item_name, + .option = vvc_metadata_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +static const enum AVCodecID vvc_metadata_codec_ids[] = { + AV_CODEC_ID_VVC, AV_CODEC_ID_NONE, +}; + +const AVBitStreamFilter ff_vvc_metadata_bsf = { + .name = "vvc_metadata", + .priv_data_size = sizeof(VVCMetadataContext), + .priv_class = &vvc_metadata_class, + .init = &vvc_metadata_init, + .close = &ff_cbs_bsf_generic_close, + .filter = &ff_cbs_bsf_generic_filter, + .codec_ids = vvc_metadata_codec_ids, +};