From patchwork Thu Jul 29 21:06:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 29116 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a5d:965a:0:0:0:0:0 with SMTP id d26csp6132283ios; Thu, 29 Jul 2021 14:07:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzvmn8sBtUKyWQ6CamrxO3/reOSl+p6PdugdihPE1yrCKXRMHvZfwTzTMVMGxb9V2i4j+p2 X-Received: by 2002:a17:906:e0cc:: with SMTP id gl12mr6262056ejb.473.1627592870077; Thu, 29 Jul 2021 14:07:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1627592870; cv=none; d=google.com; s=arc-20160816; b=dsRmGh1pfO2RbfRbvsv79cTRntJijwwK/PtCYZ2etdytRha1dLWXLp2j/xv8MEN8lk nqfiullcPs6CK48CytBwl2lmh+Hbqs9U8XP/wcWgIZiFlsDJWRKt+//hjXjGL2Zp6gJB JyRWOvbHnlo6cnXEBnW1OJCtFilTLK10uCGDCoC8Bqczzfx9V/rCOFIiXZtRATJSyqCJ YmVnSc0jXdlNa/o9GXhscsfU7J5z/WFWFBouPIPLf+jzMlrC8ZZjRKUN2UevGNIxq/mL fsQyZQpyJZzm39THfNhgWbHIb5o13Wtc//y2rIP2NvIphKZ6oxXxXycVyaVXiCyTYNV+ iEBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=eQZ5OsT7lX11j4ZqTjXCOMTYFk7Q2CwbiIle3QjDFQ8=; b=sH03ZYd1jRbHchijYQU9raLHojiD8QegGNaHG31241SQgmG9JN99GGvT5B7WZv8SLX Rjf5BLOnQx9WdxWN3NPHKisMX0dG48qrAYvtcK0EYMLUS4W68P5Hc542OCUBtubFEpIN 7SGGvYI3W1bxcD3zX+Ma+7Qi+roQhfr0X9cybQHKaM8jqhJI08JGsuA5jsT1kJJONr8c bhsX1v/ohoXW+IwfK0alLGUR4t57bet0Dv/C67pR0KqZjE5dZBFT0aKlHk+8nOVwWVtz iWoinbx/cYyRcptWhdhX8t3aN/YzyWVs5hPTFyZCuICJ7PQnbKrPW0ClbmZJuyEStp5a FY6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dZpPVAgf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b6si4054146edu.126.2021.07.29.14.07.49; Thu, 29 Jul 2021 14:07:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=dZpPVAgf; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 6C188689E84; Fri, 30 Jul 2021 00:07:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A246B687F96 for ; Fri, 30 Jul 2021 00:07:29 +0300 (EEST) Received: by mail-pl1-f176.google.com with SMTP id f13so8497337plj.2 for ; Thu, 29 Jul 2021 14:07:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=/8WPk2vJ78ZpyA7I2UXIHCDtwD2LlK3cf3ml/GHVgmU=; b=dZpPVAgfbHQ966T0KJqsjyWlHe4u9xdcRDRlBa0pYR6I33sEE8HOSidx9ClK4WI/9l N+oNn6AXdoLUKX5iA2ekMDYG59IKSBU4JZB6BOEg7qpsQItybvlMIsPsxPVXAtikodcD 5Hd8zpDNBivds2QLeQclkTUdlI4OMHMGx5uqUuM108086JI4fZ/zjVygYmZjnIuQWS8O 1o0zp8kXEDTe6rElHu2b3N0DAkPgMTW0Yw9t1whj7DNm7FQeRUNEd2vaJMdRa61WDG/1 bjuAoHNenOD0E6r3S02ktlKGXZHds7S9nnD/STl3pta/Nhpg/awcwDqc4+/f+L7CAqex GwVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/8WPk2vJ78ZpyA7I2UXIHCDtwD2LlK3cf3ml/GHVgmU=; b=hRgQFALEM9cVHRDjuuAHdYEk6BXpOU6xXwZPZVTEmq/F59i+TWqMjchIsznbwAhYo5 bUilmwHNEg3wT0OQWh6DbArBBErm80CG2CqnTiRN+ucSe1v+bAymrt8yS4pYEwBjoR2H F0SA/4jTr3WWb2xzBsmvu9YJh4r0LllAcHUDfNT1hhHIB7KeT3jnKSupRHdihqYE7jxs rxk6Jb3QSN0qKRrK38a5+7QEcm4EPsThanc+mXcAOrSmN9p8Z+5N9zCp3aUn1OSmjsrw w7A/k3guNQP3NXMHLRZqDzEwljNzULZtUeXUmuIuMc2MN5sb9dtwW0j63nL2uLI0OpMx xn6g== X-Gm-Message-State: AOAM531QOgP28tkBkP8GWQsd7ThRiteQXhilyb9a/ZoEB8KlVZCnXRsP glVBMW86JskOdKaXNZvvFixYHkXB85s= X-Received: by 2002:a17:90b:158:: with SMTP id em24mr7478370pjb.174.1627592847806; Thu, 29 Jul 2021 14:07:27 -0700 (PDT) Received: from localhost.localdomain ([191.84.244.212]) by smtp.gmail.com with ESMTPSA id c17sm4563122pfv.68.2021.07.29.14.07.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 14:07:27 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Thu, 29 Jul 2021 18:06:51 -0300 Message-Id: <20210729210651.25168-2-jamrial@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2 v2] avformat/movenc: parse h264 packets to build Sync Sample and Recovery Point tables X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: MF6hPN85ilKF Since we can't blindly trust the keyframe flag in packets and assume its contents are a valid Sync Sample, do some basic bitstream parsing to build the Sync Sample table in addition to a Random Access Recovery Point table. Suggested-by: ffmpeg@fb.com Signed-off-by: James Almer --- libavformat/movenc.c | 142 +++++++++++++++++++++++++++++++++-- libavformat/movenc.h | 1 + tests/ref/lavf-fate/h264.mp4 | 6 +- 3 files changed, 139 insertions(+), 10 deletions(-) diff --git a/libavformat/movenc.c b/libavformat/movenc.c index fedc9c0e18..01ab2f43a6 100644 --- a/libavformat/movenc.c +++ b/libavformat/movenc.c @@ -34,13 +34,17 @@ #include "avc.h" #include "libavcodec/ac3_parser_internal.h" #include "libavcodec/dnxhddata.h" +#include "libavcodec/h264.h" +#include "libavcodec/h2645_parse.h" #include "libavcodec/flac.h" #include "libavcodec/get_bits.h" +#include "libavcodec/golomb.h" #include "libavcodec/internal.h" #include "libavcodec/put_bits.h" #include "libavcodec/vc1_common.h" #include "libavcodec/raw.h" +#include "libavcodec/sei.h" #include "internal.h" #include "libavutil/avstring.h" #include "libavutil/channel_layout.h" @@ -2537,7 +2541,9 @@ static int mov_preroll_write_stbl_atoms(AVIOContext *pb, MOVTrack *track) if (!sgpd_entries) return AVERROR(ENOMEM); - av_assert0(track->par->codec_id == AV_CODEC_ID_OPUS || track->par->codec_id == AV_CODEC_ID_AAC); + av_assert0(track->par->codec_id == AV_CODEC_ID_OPUS || + track->par->codec_id == AV_CODEC_ID_AAC || + track->par->codec_id == AV_CODEC_ID_H264); if (track->par->codec_id == AV_CODEC_ID_OPUS) { for (i = 0; i < track->entry; i++) { @@ -2566,6 +2572,18 @@ static int mov_preroll_write_stbl_atoms(AVIOContext *pb, MOVTrack *track) sgpd_entries[entries].group_description_index = distance ? ++group : 0; } } + } else if (track->par->codec_id == AV_CODEC_ID_H264) { + for (i = 0; i < track->entry; i++) { + int distance = track->cluster[i].roll_distance; + if (i && distance == sgpd_entries[entries].roll_distance) { + sgpd_entries[entries].count++; + } else { + entries++; + sgpd_entries[entries].count = 1; + sgpd_entries[entries].roll_distance = distance; + sgpd_entries[entries].group_description_index = distance ? ++group : 0; + } + } } else { entries++; sgpd_entries[entries].count = track->sample_count; @@ -2639,7 +2657,9 @@ static int mov_write_stbl_tag(AVFormatContext *s, AVIOContext *pb, MOVMuxContext if (track->cenc.aes_ctr) { ff_mov_cenc_write_stbl_atoms(&track->cenc, pb); } - if (track->par->codec_id == AV_CODEC_ID_OPUS || track->par->codec_id == AV_CODEC_ID_AAC) { + if (track->par->codec_id == AV_CODEC_ID_OPUS || + track->par->codec_id == AV_CODEC_ID_AAC || + track->par->codec_id == AV_CODEC_ID_H264) { mov_preroll_write_stbl_atoms(pb, track); } return update_size(pb, pos); @@ -5150,6 +5170,107 @@ static int mov_parse_mpeg2_frame(AVPacket *pkt, uint32_t *flags) return 0; } +static int mov_parse_h264_frame(AVPacket *pkt, MOVTrack *trk, uint8_t *reformatted_data, + int size) +{ + const uint8_t *buf, *buf_end, *sei_buf = NULL; + uint32_t state = -1; + unsigned roll_distance = 0; + int nal_length_size = 0, nalsize; + int ret, idr = 0; + + if (!size) + return 0; + if (trk->vos_data && trk->vos_data[0] == 1) { + if (trk->vos_len < 5) + return 0; + nal_length_size = (trk->vos_data[4] & 0x03) + 1; + buf = pkt->data; + buf_end = pkt->data + size; + } else { + nal_length_size = 4; + buf = reformatted_data; + buf_end = reformatted_data + size; + } + + while (buf_end - buf >= 4) { + int i = 0; + nalsize = get_nalsize(nal_length_size, buf, buf_end - buf, &i, NULL); + if (nalsize < 0) + break; + state = buf[nal_length_size]; + buf += nal_length_size + 1; + nalsize--; + + switch (state & 0x1f) { + case H264_NAL_IDR_SLICE: + idr = 1; + // fall-through + case H264_NAL_SLICE: + goto end; + case H264_NAL_SEI: { + int sei_size; + + sei_buf = ff_nal_unit_extract_rbsp(buf, nalsize, &sei_size, 0); + if (!sei_buf) + goto end; + + i = 0; + do { + GetBitContext gb; + int payload_type = 0, payload_size = 0; + + do { + if (sei_size - i < 1) + goto end; + payload_type += sei_buf[i]; + } while (sei_buf[i++] == 255); + do { + if (sei_size - i < 1) + goto end; + payload_size += sei_buf[i]; + } while (sei_buf[i++] == 255); + + if (payload_size > sei_size - i) + goto end; + + ret = init_get_bits8(&gb, sei_buf + i, payload_size); + if (ret < 0) + goto end; + + switch (payload_type) { + case SEI_TYPE_RECOVERY_POINT: + roll_distance = get_ue_golomb_long(&gb); + break; + default: + break; + } + i += payload_size; + } while (sei_size - i > 0 && sei_buf[i] != 0x80); + av_freep(&sei_buf); + break; + } + default: + break; + } + buf += nalsize; + } + +end: + if (roll_distance != (int16_t)roll_distance) + roll_distance = 0; + trk->cluster[trk->entry].roll_distance = roll_distance; + + if (idr) { + trk->cluster[trk->entry].flags |= MOV_SYNC_SAMPLE; + trk->has_keyframes++; + } + + av_freep(&sei_buf); + + return 0; +} + static void mov_parse_vc1_frame(AVPacket *pkt, MOVTrack *trk) { const uint8_t *start, *next, *end = pkt->data + pkt->size; @@ -5638,13 +5759,17 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt) avio_write(pb, reformatted_data, size); } else { if (trk->cenc.aes_ctr) { - size = ff_mov_cenc_avc_parse_nal_units(&trk->cenc, pb, pkt->data, size); - if (size < 0) { - ret = size; + ret = ff_avc_parse_nal_units_buf(pkt->data, &reformatted_data, &size); + if (ret < 0) + return ret; + ret = ff_mov_cenc_avc_write_nal_units(s, &trk->cenc, 4, pb, reformatted_data, size); + if (ret < 0) goto err; - } } else { - size = ff_avc_parse_nal_units(pb, pkt->data, pkt->size); + ret = ff_avc_parse_nal_units_buf(pkt->data, &reformatted_data, &size); + if (ret < 0) + return ret; + avio_write(pb, reformatted_data, size); } } } else if (par->codec_id == AV_CODEC_ID_HEVC && trk->vos_len > 6 && @@ -5740,6 +5865,7 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt) trk->cluster[trk->entry].entries = samples_in_chunk; trk->cluster[trk->entry].dts = pkt->dts; trk->cluster[trk->entry].pts = pkt->pts; + trk->cluster[trk->entry].roll_distance = 0; if (!trk->entry && trk->start_dts != AV_NOPTS_VALUE) { if (!trk->frag_discont) { /* First packet of a new fragment. We already wrote the duration @@ -5821,6 +5947,8 @@ int ff_mov_write_packet(AVFormatContext *s, AVPacket *pkt) if (par->codec_id == AV_CODEC_ID_VC1) { mov_parse_vc1_frame(pkt, trk); + } else if (par->codec_id == AV_CODEC_ID_H264 && !TAG_IS_AVCI(trk->tag)) { + mov_parse_h264_frame(pkt, trk, reformatted_data, size); } else if (par->codec_id == AV_CODEC_ID_TRUEHD) { mov_parse_truehd_frame(pkt, trk); } else if (pkt->flags & AV_PKT_FLAG_KEY) { diff --git a/libavformat/movenc.h b/libavformat/movenc.h index af1ea0bce6..73bf73ce8f 100644 --- a/libavformat/movenc.h +++ b/libavformat/movenc.h @@ -56,6 +56,7 @@ typedef struct MOVIentry { #define MOV_PARTIAL_SYNC_SAMPLE 0x0002 #define MOV_DISPOSABLE_SAMPLE 0x0004 uint32_t flags; + int16_t roll_distance; AVProducerReferenceTime prft; } MOVIentry; diff --git a/tests/ref/lavf-fate/h264.mp4 b/tests/ref/lavf-fate/h264.mp4 index a9c3823c2c..c08ee4c7ae 100644 --- a/tests/ref/lavf-fate/h264.mp4 +++ b/tests/ref/lavf-fate/h264.mp4 @@ -1,3 +1,3 @@ -fe299ea5205b71a48281f917b1256a5d *tests/data/lavf-fate/lavf.h264.mp4 -547928 tests/data/lavf-fate/lavf.h264.mp4 -tests/data/lavf-fate/lavf.h264.mp4 CRC=0x9da2c999 +2812f617314d23474fcb23898b8a56ab *tests/data/lavf-fate/lavf.h264.mp4 +548084 tests/data/lavf-fate/lavf.h264.mp4 +tests/data/lavf-fate/lavf.h264.mp4 CRC=0x396f0829