From patchwork Sat Jan 9 07:34:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nuo Mi X-Patchwork-Id: 24859 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 3653244AF8E for ; Sat, 9 Jan 2021 09:35:24 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1CEB868A71F; Sat, 9 Jan 2021 09:35:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 10ED068A755 for ; Sat, 9 Jan 2021 09:35:22 +0200 (EET) Received: by mail-pf1-f177.google.com with SMTP id h186so7746196pfe.0 for ; Fri, 08 Jan 2021 23:35:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wClPzTSLRsJMiPJzf004FtGaIhxMtXHzJTwzWeDxPgY=; b=fVDqywLP99GUKpnSImc5E3NtH4A/uRJqcREaq5fLdfWRxUv2l84MCi5xfULP801iJ9 Qz0UZS6YeP8oTPITZh5z7l4u8XeNR/ISHSsiQORxncsD2D4Gmwfx5as9aDhtHsUC5s5g U2rbyQkSZgH7GDqXef/aiaiS5kzmmyhcxvmI7keC/4zti69SsgsUxB5zmng/KOXYIex3 JfcXKnnzdymdgXMjzEOfUVRq4SJMIkZKxJ+ucZN1Hv8W++E/zcLk7ILOAwSI9Zo040kR KHAPOgbj988wgijLzNdUlcBn7mLP6IYQHOc9Sclm8HDVuXBmsjKMhJ1t3xcs+FW+q0ci alKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wClPzTSLRsJMiPJzf004FtGaIhxMtXHzJTwzWeDxPgY=; b=pWWvPf0wF2EMQUZg88d6wpAt5eSdPul5rceA9bGxhxTFKaGsIvrejKDzKfyoFltWoa 7orx53uGFxWibp93qWmLwo9sErpRXA3Lgh3NYsLROouI7EP6MUFg6ekQf1Dq8mxArNBW 5gTy0zJwbkpW1cbE7LjsvbIC6/9B0P1H33jX4LPT3W10Y1v4St0Hp5XdmukkQSXBd+Q3 MimiYDHhiWKD5FV4yxW4k9wPMWptcOWa9dqDl0PG+9OWeHPjCofK0venBkVxvWDG16Zl gO8KirN4616qSTgm9P/FG8fthYEslJM85gUIsM7nJjjlPl/dT73FKOMB79bspObap/tX utJg== X-Gm-Message-State: AOAM530OXFLzFCDRsQkADUqQ0y5eNKQNhpde/EXad/CffsKdY9DkE66n i8rHEKf46DfRzbuMolfbhn/d3xC35dUXfQ== X-Google-Smtp-Source: ABdhPJyPIdDfrCW/Sy0/Ommx906CVIBG1lSWUibd2oXJQS2uOFlKkETM6ODBkm1kLUTAE7CxMQi8nA== X-Received: by 2002:a62:4d03:0:b029:1ac:6159:4572 with SMTP id a3-20020a624d030000b02901ac61594572mr7470384pfb.10.1610177721021; Fri, 08 Jan 2021 23:35:21 -0800 (PST) Received: from Guangxin-PC.localdomain (23.83.245.51.16clouds.com. [23.83.245.51]) by smtp.gmail.com with ESMTPSA id m15sm11640120pfa.72.2021.01.08.23.35.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jan 2021 23:35:20 -0800 (PST) From: Nuo Mi To: ffmpeg-devel@ffmpeg.org Date: Sat, 9 Jan 2021 15:34:20 +0800 Message-Id: <20210109073421.23721-11-nuomi2021@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210109073421.23721-1-nuomi2021@gmail.com> References: <20210109073421.23721-1-nuomi2021@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2 10/11] avcodec: add vvc metadata bsf X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Nuo Mi Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" use following command to test: ffmpeg -i in.bin -c:v copy -bsf vvc_metadata -f vvc out.bin 79.11%(197/249) can bit match with original clips 6.83%(17/249) are not bit match, the original clips has redundant emulation prevent bytes other all failed since vvdec failed to decode the first frame --- configure | 1 + libavcodec/Makefile | 1 + libavcodec/bitstream_filters.c | 1 + libavcodec/vvc_metadata_bsf.c | 243 +++++++++++++++++++++++++++++++++ 4 files changed, 246 insertions(+) create mode 100644 libavcodec/vvc_metadata_bsf.c diff --git a/configure b/configure index 5ff743d9c2..b41f2af151 100755 --- a/configure +++ b/configure @@ -3184,6 +3184,7 @@ mjpeg2jpeg_bsf_select="jpegtables" mpeg2_metadata_bsf_select="cbs_mpeg2" trace_headers_bsf_select="cbs" vp9_metadata_bsf_select="cbs_vp9" +vvc_metadata_bsf_select="cbs_h266" # external libraries aac_at_decoder_deps="audiotoolbox" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 30e9a92e9f..d4cd64e43c 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1166,6 +1166,7 @@ OBJS-$(CONFIG_VP9_METADATA_BSF) += vp9_metadata_bsf.o OBJS-$(CONFIG_VP9_RAW_REORDER_BSF) += vp9_raw_reorder_bsf.o OBJS-$(CONFIG_VP9_SUPERFRAME_BSF) += vp9_superframe_bsf.o OBJS-$(CONFIG_VP9_SUPERFRAME_SPLIT_BSF) += vp9_superframe_split_bsf.o +OBJS-$(CONFIG_VVC_METADATA_BSF) += vvc_metadata_bsf.o # thread libraries OBJS-$(HAVE_LIBC_MSVCRT) += file_open.o diff --git a/libavcodec/bitstream_filters.c b/libavcodec/bitstream_filters.c index b26d6a910e..001a7bb3a4 100644 --- a/libavcodec/bitstream_filters.c +++ b/libavcodec/bitstream_filters.c @@ -60,6 +60,7 @@ extern const AVBitStreamFilter ff_vp9_metadata_bsf; extern const AVBitStreamFilter ff_vp9_raw_reorder_bsf; extern const AVBitStreamFilter ff_vp9_superframe_bsf; extern const AVBitStreamFilter ff_vp9_superframe_split_bsf; +extern const AVBitStreamFilter ff_vvc_metadata_bsf; #include "libavcodec/bsf_list.c" diff --git a/libavcodec/vvc_metadata_bsf.c b/libavcodec/vvc_metadata_bsf.c new file mode 100644 index 0000000000..a7deced47c --- /dev/null +++ b/libavcodec/vvc_metadata_bsf.c @@ -0,0 +1,243 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/common.h" +#include "libavutil/opt.h" + +#include "bsf.h" +#include "bsf_internal.h" +#include "cbs.h" +#include "cbs_h266.h" +#include "vvc.h" + +enum { + PASS, + INSERT, + REMOVE, +}; + +typedef struct VVCMetadataContext { + const AVClass *class; + + CodedBitstreamContext *input; + CodedBitstreamContext *output; + CodedBitstreamFragment access_unit; + + H266RawAUD aud_nal; + + int aud; + +} VVCMetadataContext; + +static int vvc_metadata_update_side_data(AVBSFContext *bsf, AVPacket *pkt) +{ + VVCMetadataContext *ctx = bsf->priv_data; + CodedBitstreamFragment *au = &ctx->access_unit; + uint8_t *side_data; + int side_data_size; + int err; + + side_data = av_packet_get_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, + &side_data_size); + if (!side_data_size) + return 0; + + err = ff_cbs_read(ctx->input, au, side_data, side_data_size); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to read extradata from packet side data.\n"); + return err; + } + + err = ff_cbs_write_fragment_data(ctx->output, au); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write extradata into packet side data.\n"); + return err; + } + + side_data = av_packet_new_side_data(pkt, AV_PKT_DATA_NEW_EXTRADATA, au->data_size); + if (!side_data) + return AVERROR(ENOMEM); + memcpy(side_data, au->data, au->data_size); + + ff_cbs_fragment_reset(au); + + return 0; +} + +static int vvc_metadata_filter(AVBSFContext *bsf, AVPacket *pkt) +{ + VVCMetadataContext *ctx = bsf->priv_data; + CodedBitstreamFragment *au = &ctx->access_unit; + int err, i; + + err = ff_bsf_get_packet_ref(bsf, pkt); + if (err < 0) + return err; + + err = vvc_metadata_update_side_data(bsf, pkt); + if (err < 0) + goto fail; + + err = ff_cbs_read_packet(ctx->input, au, pkt); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to read packet.\n"); + goto fail; + } + + if (au->nb_units == 0) { + av_log(bsf, AV_LOG_ERROR, "No NAL units in packet.\n"); + err = AVERROR_INVALIDDATA; + goto fail; + } + + // If an AUD is present, it must be the first NAL unit. + if (au->units[0].type == VVC_AUD_NUT) { + if (ctx->aud == REMOVE) + ff_cbs_delete_unit(au, 0); + } else { + if (ctx->aud == INSERT) { + H266RawAUD *aud = &ctx->aud_nal; + int pic_type = 0, temporal_id = 8, layer_id = 0; + CodedBitstreamH266Context *h266 = ctx->input->priv_data; + for (i = 0; i < au->nb_units; i++) { + const H266RawNALUnitHeader *nal = au->units[i].content; + if (!nal) + continue; + if (nal->nuh_temporal_id_plus1 < temporal_id + 1) + temporal_id = nal->nuh_temporal_id_plus1 - 1; + + if (au->units[i].type <= VVC_RSV_IRAP_11) { + const H266RawSlice *slice = au->units[i].content; + layer_id = nal->nuh_layer_id; + if (slice->header.sh_slice_type == VVC_SLICE_TYPE_B && + pic_type < 2) + pic_type = 2; + if (slice->header.sh_slice_type == VVC_SLICE_TYPE_P && + pic_type < 1) + pic_type = 1; + } + } + + aud->nal_unit_header = (H266RawNALUnitHeader) { + .nal_unit_type = VVC_AUD_NUT, + .nuh_layer_id = layer_id, + .nuh_temporal_id_plus1 = temporal_id + 1, + }; + aud->aud_pic_type = pic_type; + aud->aud_irap_or_gdr_flag = h266->active_ph->ph_gdr_or_irap_pic_flag; + + err = ff_cbs_insert_unit_content(au, 0, VVC_AUD_NUT, aud, NULL); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to insert AUD.\n"); + goto fail; + } + } + } + + err = ff_cbs_write_packet(ctx->output, pkt, au); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write packet.\n"); + goto fail; + } + +fail: + ff_cbs_fragment_reset(au); + + if (err < 0) + av_packet_unref(pkt); + + return err; +} + +static int vvc_metadata_init(AVBSFContext *bsf) +{ + VVCMetadataContext *ctx = bsf->priv_data; + CodedBitstreamFragment *au = &ctx->access_unit; + int err; + + err = ff_cbs_init(&ctx->input, AV_CODEC_ID_VVC, bsf); + if (err < 0) + return err; + err = ff_cbs_init(&ctx->output, AV_CODEC_ID_VVC, bsf); + if (err < 0) + return err; + + if (bsf->par_in->extradata) { + err = ff_cbs_read_extradata(ctx->input, au, bsf->par_in); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to read extradata.\n"); + goto fail; + } + + err = ff_cbs_write_extradata(ctx->output, bsf->par_out, au); + if (err < 0) { + av_log(bsf, AV_LOG_ERROR, "Failed to write extradata.\n"); + goto fail; + } + } + +fail: + ff_cbs_fragment_reset(au); + return err; +} + +static void vvc_metadata_close(AVBSFContext *bsf) +{ + VVCMetadataContext *ctx = bsf->priv_data; + + ff_cbs_fragment_free(&ctx->access_unit); + ff_cbs_close(&ctx->input); + ff_cbs_close(&ctx->output); +} + +#define OFFSET(x) offsetof(VVCMetadataContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_BSF_PARAM) +static const AVOption vvc_metadata_options[] = { + { "aud", "Access Unit Delimiter NAL units", + OFFSET(aud), AV_OPT_TYPE_INT, + { .i64 = PASS }, PASS, REMOVE, FLAGS, "aud" }, + { "pass", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = PASS }, .flags = FLAGS, .unit = "aud" }, + { "insert", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = INSERT }, .flags = FLAGS, .unit = "aud" }, + { "remove", NULL, 0, AV_OPT_TYPE_CONST, + { .i64 = REMOVE }, .flags = FLAGS, .unit = "aud" }, + + { NULL } +}; + +static const AVClass vvc_metadata_class = { + .class_name = "vvc_metadata_bsf", + .item_name = av_default_item_name, + .option = vvc_metadata_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +static const enum AVCodecID vvc_metadata_codec_ids[] = { + AV_CODEC_ID_VVC, AV_CODEC_ID_NONE, +}; + +const AVBitStreamFilter ff_vvc_metadata_bsf = { + .name = "vvc_metadata", + .priv_data_size = sizeof(VVCMetadataContext), + .priv_class = &vvc_metadata_class, + .init = &vvc_metadata_init, + .close = &vvc_metadata_close, + .filter = &vvc_metadata_filter, + .codec_ids = vvc_metadata_codec_ids, +};