From patchwork Sun Feb 12 23:06:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Coiner X-Patchwork-Id: 40374 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:5494:b0:bf:7b3a:fd32 with SMTP id i20csp3045653pzk; Sun, 12 Feb 2023 15:06:24 -0800 (PST) X-Google-Smtp-Source: AK7set+aLiiPZRj3tr7A3dVqbVJ9YUDYsAzLBL3fhxQJt7XhrDu0GJDtDE72YT2RIColuiyczMB1 X-Received: by 2002:a17:906:3ec1:b0:88f:87f0:c919 with SMTP id d1-20020a1709063ec100b0088f87f0c919mr22855900ejj.64.1676243183866; Sun, 12 Feb 2023 15:06:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676243183; cv=none; d=google.com; s=arc-20160816; b=L2T+46xjfKzpufPg3bepaDUZ6djHcXQ09/K3hSAkk8G4tvI5X9i1RxxoOhS5gAUA7V qTaQiJIImkskTqOa3vWWUCkSgdJkNBtbJtoujX8lqGOyXL+OtmOWR07Ms/j4XmrXnzmV zurlC9Hkn4EAgnENvcWfe5Xyi7CK42CpXYuau+WixehPKegs4Av8TBU8EQDmUSEuRL+6 MnEq/YovcEMSmNL9OUBf1mvTPJnohdysHQdAN6KZoZpzEyWnQhLa6BOcmYCarkkp4NVZ m2r6vdq7xJLqvzisio2cjixsTRhHW6WWFE/Y24gDIIANtjlgD5BSbuDgwSZiTdxB/8a5 o+4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=XoTbZf3pywqKSN6Wb1XiBL5Rn0qoRE56Wg4XUvvbIYk=; b=aGJOLo/aLxWcPeWHqUGKBe4tNdEqTxxaXHSjWajcKYAkaRuXUyfPGV7qilMFBJgulb L5niYLDf/HiBh45aNovQpbjLMtVcmSHbGqqn4BIlnOJ5yvN4+CLF0ryJXY+7HoHMp6HD lS00raoPCV2QH37ZL/JKuoEyIfYDtRGy2ysE9kdTgJIXxtzz6byw5ilTiDPWaFv7aVSu 9YWGsQ2cO7kMPBNv8sZcYfaozSh+u5A1QOLPRgJAvgTPplm4M+wQQvUrIrXHjscKMU6V ohDaZyKa3RmBZV/DfRI2JYxcfhycL5WdUokKWX2nu8/rzvptIcwdiA30EhYKmwsQ1+A9 ahlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=khiJiAK4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mr17-20020a170907829100b008b103d41fd8si2496433ejc.806.2023.02.12.15.06.23; Sun, 12 Feb 2023 15:06:23 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20210112 header.b=khiJiAK4; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CC36E68BDE5; Mon, 13 Feb 2023 01:06:19 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AFEAC68BA81 for ; Mon, 13 Feb 2023 01:06:12 +0200 (EET) Received: by mail-qv1-f43.google.com with SMTP id j5so5190632qvi.3 for ; Sun, 12 Feb 2023 15:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1676243171; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=q6d37sP51VlnzAqEelDOhGY7x8YtjS6kVmER0Vm59FA=; b=khiJiAK4TRjVI45gf0PfNWDJceUVH+4t6iy90DfIUMoAhzdwhxde1Nco7hQTkxlvp+ e71gqHqqsJbdgEjY42/jePsHqPLRLp3Y+4mWe7KFNaTolXjHoGprTHM+VTbTo0bJ7qpy Nm1LpFZyY4rsq6YL5vEo2nydsxcrYQ98QeK8aKIQua3xbPxqkjAjzUX/O7dIR9gdvxzC raSGOntfTFUSVrQDQUnwj+UbmfI9kuzlcxKpoq9Ge4DEyfM5H2oycjqARn8/fBjvwSm5 hqXvHtAq6d3b5KlTA7njzYGFLdaa2K4IN9g0zhhzyMneNnbkqdploi21s0hlywVJuygp 1T0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676243171; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=q6d37sP51VlnzAqEelDOhGY7x8YtjS6kVmER0Vm59FA=; b=7q6peoun4RxLthn2FCrlqhG3lA1zlwmo4A3kIBFNVj0o53GwbCBL2buq2dikFpCOB2 UDUoukOIlKw4tdbyPktqs/z1IM5u/h1nelcU1h+TxwJVy3nOTqZ7iYtKyZVO8AYw6H+w EN8F+++HYld19Kj3tYkufl18yhV4DX9Wm/Mn2QOh9N+X0a+YjqEgMuOxh8YYNQUpR5MR jU87ywoIwS0poWar+7AcODqpGBkug2zzpv76z/vYJ1tTwddzJ1CbaSsbmW8nXP6lNID/ muq42LA2uXk+uLrP2R82TqSgsBVN4iqLvE3FBY+paoGEwlz8hrlbdXREI1yrjvhgU6i+ 97uA== X-Gm-Message-State: AO0yUKUdKptJK6KF2VeIFK9oYWbd55DHUM4HTsxQi9GFAWw239clkku6 UNBl7djR3OkVwYfXD9DRllG3SQ6jETo= X-Received: by 2002:a0c:f1d0:0:b0:539:4aa4:3422 with SMTP id u16-20020a0cf1d0000000b005394aa43422mr31604148qvl.0.1676243170951; Sun, 12 Feb 2023 15:06:10 -0800 (PST) Received: from jcoiner3.corp.google.com ([2620:15c:93:200:3823:7d8b:7a22:f926]) by smtp.gmail.com with ESMTPSA id d64-20020a37b443000000b006ff8a122a1asm8434248qkf.78.2023.02.12.15.06.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Feb 2023 15:06:10 -0800 (PST) From: John Coiner To: ffmpeg-devel@ffmpeg.org Date: Sun, 12 Feb 2023 18:06:07 -0500 Message-Id: <20230212230607.180234-1-jpcoiner@gmail.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH v2] libavformat/mpegtsenc.c: correctly re-emit extradata ahead of IDR pictures X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: John Coiner , cus@passwd.hu Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 2xDIznrkKdq0 This is v2 of the patch for https://trac.ffmpeg.org/ticket/10148. It implements the handling described at http://ffmpeg.org/pipermail/ffmpeg-devel/2023-February/306542.html, that is: * If we receive [AUD][IDR], we emit [AUD][SPS][PPS][IDR] * If we receive [SEI][AUD][IDR], we emit [AUD][SPS][PPS][SEI][IDR] This is speculative; it would be good to hear from Marton or others about whether this is the right handling. The other possible handling would be to simply prepend the extradata to the existing frame's bitstream without relocating the AUD to be in front. So if we received: [AUD][IDR] We would emit: [SPS][PPS][AUD][IDR] That's not quite compliant with the H.264 spec, which says an AUD shall be the first NAL in a frame when it's present. I doubt real decoders care and it would be simpler to implement. Let me know if you'd prefer the simpler handling. --- libavformat/mpegtsenc.c | 55 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 49 insertions(+), 6 deletions(-) diff --git a/libavformat/mpegtsenc.c b/libavformat/mpegtsenc.c index 00ad426086..a4a2d8cdaf 100644 --- a/libavformat/mpegtsenc.c +++ b/libavformat/mpegtsenc.c @@ -1835,6 +1835,13 @@ static int opus_get_packet_samples(AVFormatContext *s, AVPacket *pkt) return duration; } +// Copies `size_bytes` from `source` to `dest`. +// Returns a pointer to the next destination address after the range written. +static uint8_t* append(uint8_t* dest, const uint8_t* source, int size_bytes) { + memcpy(dest, source, size_bytes); + return dest + size_bytes; +} + static int mpegts_write_packet_internal(AVFormatContext *s, AVPacket *pkt) { AVStream *st = s->streams[pkt->stream_index]; @@ -1877,6 +1884,7 @@ static int mpegts_write_packet_internal(AVFormatContext *s, AVPacket *pkt) if (st->codecpar->codec_id == AV_CODEC_ID_H264) { const uint8_t *p = buf, *buf_end = p + size; + const uint8_t *found_aud = NULL, *found_aud_end = NULL; uint32_t state = -1; int extradd = (pkt->flags & AV_PKT_FLAG_KEY) ? st->codecpar->extradata_size : 0; int ret = ff_check_h264_startcode(s, st, pkt); @@ -1886,17 +1894,34 @@ static int mpegts_write_packet_internal(AVFormatContext *s, AVPacket *pkt) if (extradd && AV_RB24(st->codecpar->extradata) > 1) extradd = 0; + // Ensure that all pictures are prefixed with an AUD, and that + // IDR pictures are also prefixed with SPS and PPS. SPS and PPS + // are assumed to be available in 'extradata' if not found in-band. do { p = avpriv_find_start_code(p, buf_end, &state); av_log(s, AV_LOG_TRACE, "nal %"PRId32"\n", state & 0x1f); - if ((state & 0x1f) == 7) + if ((state & 0x1f) == 7) // SPS NAL extradd = 0; - } while (p < buf_end && (state & 0x1f) != 9 && - (state & 0x1f) != 5 && (state & 0x1f) != 1); - - if ((state & 0x1f) != 5) + if ((state & 0x1f) == 9) { // AUD NAL + found_aud = p - 4; // start of the 0x000001 start code. + found_aud_end = p + 1; // first byte past the AUD. + + if (found_aud < buf) + found_aud = buf; + if (buf_end < found_aud_end) + found_aud_end = buf_end; + } + } while (p < buf_end + && (state & 0x1f) != 5 // IDR picture + && (state & 0x1f) != 1 // non-IDR picture + && (extradd > 0 || !found_aud)); + if ((state & 0x1f) != 5) { + // Did not find an IDR picture; do not emit extradata. extradd = 0; - if ((state & 0x1f) != 9) { // AUD NAL + } + + if (!found_aud) { + // Prefix 'buf' with the missing AUD, and extradata if needed. data = av_malloc(pkt->size + 6 + extradd); if (!data) return AVERROR(ENOMEM); @@ -1907,6 +1932,24 @@ static int mpegts_write_packet_internal(AVFormatContext *s, AVPacket *pkt) data[5] = 0xf0; // any slice type (0xe) + rbsp stop one bit buf = data; size = pkt->size + 6 + extradd; + } else if (extradd != 0) { + // Move the AUD up to the beginning of the frame, where the H.264 + // spec requires it to appear. Emit the extradata after it. + const int aud_size = found_aud_end - found_aud; + const int new_pkt_size = pkt->size + 1 + extradd; + uint8_t *pos; + data = av_malloc(new_pkt_size); + if (!data) + return AVERROR(ENOMEM); + AV_WB8(data, 0x0); + pos = data + 1; + pos = append(pos, found_aud, aud_size); + pos = append(pos, st->codecpar->extradata, extradd); + pos = append(pos, pkt->data, found_aud - pkt->data); + pos = append(pos, found_aud_end, buf_end - found_aud_end); + av_assert0(data + new_pkt_size == pos); + buf = data; + size = new_pkt_size; } } else if (st->codecpar->codec_id == AV_CODEC_ID_AAC) { if (pkt->size < 2) {