From patchwork Fri May 19 16:41:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 41724 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:ba91:b0:105:feb:71f2 with SMTP id fb17csp1130390pzb; Fri, 19 May 2023 01:42:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4MXxng55E5Me/ZCA3UB9UMsg21sAjeEe3lqfu2XQbaW8LoaqtYwXglsLH7yCZ0hLHsQ6bL X-Received: by 2002:a17:907:6d26:b0:957:2e48:5657 with SMTP id sa38-20020a1709076d2600b009572e485657mr878717ejc.68.1684485737836; Fri, 19 May 2023 01:42:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684485737; cv=none; d=google.com; s=arc-20160816; b=mbyQkBAesB5WzkedUVWW9OsgtA6+6SLwKJ50IBec9Lo/eAmMU5nrl7fwpPie4G4CDF xZHLlfZbjjiWnYsyliNS9mpBcJ8LNdxVSzYxMuD1TTMl8vl/WpoUrjR8F+LrIqFST8Ze GSudVMi7vpqZzeCnE7WVvlIxu5LRx+g+8Sw/VfRvx/sx8uLVU5D8stwPf6n4rok33uZ+ X59IC19BeqXkEq+XNDU0yZl3CbxntM7rTt6lR/RXhO/jzLAwlUQT4b8MbgK71Wxz1ElK jcdud30jXyu80xdvZ+SRrQw95RSdjPq7eG19TCnYmQUnZAU21C2ROOkqHAEvRKnnddFg h5dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to:date :to:from:message-id:dkim-signature:delivered-to; bh=tHN2u8JMx1EUYBBUUf6ugn4hS/mxpiaiftLXtyKRARk=; b=H+SHhwNuEWVwAXAOHT8BmEl7BqHNIA+sNwjU1KdOcmzAIgZE5BDf+5ahNVZ04d9rir OYmzdvZcugX2cNwcq9V7jtbjfJGBEU0EQ7ReNWOg1pe9NJXJRGeUowUPcjzKUCaso5n4 zJMmjOaWV2L1CmE3n4SIO8c/DhxlP+a+xTI1jwQX79O+ML863cW38XaeD4QCVvwx8o6x BcoxiP7zlwL3S85L90inKvt05zw6h9bWyNilVPcA0JpHzEp/Vh+RKG4ggKhd7mFMO9cA zwJ7AUSuX9WQwipSLqvi3qRsKNxAQbhV7NbKJBcXx+vSmQhQkmFDZwZc0yDBkjzWIxDH 6smg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="qWAsQ8+/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id mj3-20020a170906af8300b0094f1eee5a9dsi2640588ejb.555.2023.05.19.01.42.17; Fri, 19 May 2023 01:42:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="qWAsQ8+/"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 284E468C178; Fri, 19 May 2023 11:41:48 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out162-62-57-64.mail.qq.com (out162-62-57-64.mail.qq.com [162.62.57.64]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 429E368ADCE for ; Fri, 19 May 2023 11:41:37 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1684485689; bh=6uEL8MCQaN7adfQeSB0qUt90oiFuebfwuv9AbaORl/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=qWAsQ8+/TXwUhq7KblP8wi9IYo+LDFomRNfoUn7apiIdKbgmgo7BHK54dbvzWJgSc Rf/n6mjSwRsMOUHXYFXZ7U2f+LR4ympwws85j5xr4UoFKsHMZHjvTGn0S9ODoX0aE0 hpQEDgtY1uJbWd6ZLCSP6M+d1Xkgq7qaqMZSRqzA= Received: from localhost.localdomain ([119.147.10.201]) by newxmesmtplogicsvrszc2-0.qq.com (NewEsmtp) with SMTP id A5B2749A; Fri, 19 May 2023 16:41:27 +0800 X-QQ-mid: xmsmtpt1684485688t9pvrocsc Message-ID: X-QQ-XMAILINFO: MB5+LsFw85NoStkxfg1JR/6sZSpKwexhXHqxDEFCfr/gEMxlIhgj/mjt0wC0oy lOoTWnkfgSvD5Ih6LBD68A09qT+VSZZSLeWd7pTLBRb27JA9dY799ovbtMBBi+Ei36JgX1FmsJW/ XvQPi3WKbHFsQGfB843SruwxhxJXuCfYIMedChLQKJa/sYaO+lGLfYDd7+rxI1Mnk/nrWkAqo8pI g18RFcTCcKeH2ctPC3xB8JfdUxq8caOwC3AOEa7LJX4KSYnkJExzSHccSE5UN1WWsPS4MK+nSruO EeffdO+xl9XtGlOkKt3aJkZSm243b336YUz2aomjWXiQTvYwvIm2YGuZ8o7KUAlvQAen6bRoQbVl 9/5rB7GT78ghTiOGMfhhbcL1MQDAtFBtBIP9Siq2OJhwaKsjksk2RhKjJ8QlBxSxiEpvvRN4HoK2 bBGvthCvFCINEM/vcQrM9jbDIjRG/9JFSGz77yWelxsmDFf1rg3kb/yvzsvUKKk2lMZ07kgXCTNe Jb/WpNQrorzzTgDibPD164CZ3WgdCmuMucMeXB1lIptGQiir+8TR5TyEMsWpcNrdXkNgnseyqhkz K+d64Zk/R6XWPsNZtzIwGlaiT48/6fttFkLjV8T4FWw/iLJ6Uq3lBW6mpK04QuAKN3UWuB/oKsDD pBMAYuOqLWbSR0Yoo5f9Lo17JrhPeQsBzhvbdlSIa2A2GZ0EPrQ5Qc/0EQIabCUyQ27VSPNsxWgW ojTbDx7SRFn6TxawxRNvonaEgSlyZaYPWdJnwC7u+/Wt1TjeCJB32u/VH6dbNFxeRcX9Sfc65xQ1 tlr57jFqhGvLyQr2jdVRwdIaPI/rD4NndgfBlkHWRyV+y8aNM+nSt0bpT2fxbr+I6PIj5ipFCqWK DI5tivADDhKjWQqPo44vK8ooCmyrSZyYtQSeUvZaZl0pkJGg09hU5FQ2zDK/XFIYGk+aE7UMa0eC UzDo2tzX5deIf4Xxw3PRlmLPN5HE7qL1bBxngpoBlQMZw/uDTdUf7ZZUSKA9ecnuPcQoPiqwa6c3 WECSK3Dhsh8K1Khi+nWM+vjAu1Low= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sat, 20 May 2023 00:41:31 +0800 X-OQ-MSGID: <20230519164133.3278597-3-quinkblack@foxmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230519164133.3278597-1-quinkblack@foxmail.com> References: <20230519164133.3278597-1-quinkblack@foxmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 3/5] avcodec/h264_mp4toannexb_bsf: fix missing PS before IDR frames X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: jndADAbcJgYv From: Zhao Zhili If there is a single group of SPS/PPS before an IDR frame, but no SPS/PPS after that, we will miss the chance to reset idr_sps_seen/idr_pps_seen. No SPS/PPS are inserted afterwards. This patch saves in-band SPS/PPS and insert them before IDR frames when necessary. --- sample file h264/ps_prefix_first_idr.mp4 https://drive.google.com/file/d/1GEImeE-Wx1T0veQClSuasEJbs4fssMo9/view?usp=sharing libavcodec/h264_mp4toannexb_bsf.c | 83 ++++++++++++++++++++++++++++--- tests/fate/h264.mak | 8 ++- 2 files changed, 84 insertions(+), 7 deletions(-) diff --git a/libavcodec/h264_mp4toannexb_bsf.c b/libavcodec/h264_mp4toannexb_bsf.c index 846671abb6..4073c780c5 100644 --- a/libavcodec/h264_mp4toannexb_bsf.c +++ b/libavcodec/h264_mp4toannexb_bsf.c @@ -36,6 +36,8 @@ typedef struct H264BSFContext { uint8_t *pps; int sps_size; int pps_size; + unsigned sps_buf_size; + unsigned pps_buf_size; uint8_t length_size; uint8_t new_idr; uint8_t idr_sps_seen; @@ -130,16 +132,33 @@ pps: memset(out + total_size, 0, padding); if (pps_offset) { - s->sps = out; + uint8_t *sps; + s->sps_size = pps_offset; + sps = av_fast_realloc(s->sps, &s->sps_buf_size, s->sps_size); + if (!sps) { + av_free(out); + return AVERROR(ENOMEM); + } + s->sps = sps; + memcpy(s->sps, out, s->sps_size); } else { av_log(ctx, AV_LOG_WARNING, "Warning: SPS NALU missing or invalid. " "The resulting stream may not play.\n"); } if (pps_offset < total_size) { - s->pps = out + pps_offset; + uint8_t *pps; + s->pps_size = total_size - pps_offset; + pps = av_fast_realloc(s->pps, &s->pps_buf_size, s->pps_size); + if (!pps) { + av_freep(&s->sps); + av_free(out); + return AVERROR(ENOMEM); + } + s->pps = pps; + memcpy(s->pps, out + pps_offset, s->pps_size); } else { av_log(ctx, AV_LOG_WARNING, "Warning: PPS NALU missing or invalid. " @@ -153,6 +172,35 @@ pps: return length_size; } +static int h264_mp4toannexb_save_ps(uint8_t **dst, int *dst_size, + unsigned *dst_buf_size, + const uint8_t *nal, uint32_t nal_size, + int first) +{ + static const uint8_t nalu_header[4] = { 0, 0, 0, 1 }; + const int start_code_size = sizeof(nalu_header); + uint8_t *ptr; + uint32_t size; + + if (first) + size = 0; + else + size = *dst_size; + + ptr = av_fast_realloc(*dst, dst_buf_size, size + nal_size + start_code_size); + if (!ptr) + return AVERROR(ENOMEM); + + memcpy(ptr + size, nalu_header, start_code_size); + size += start_code_size; + memcpy(ptr + size, nal, nal_size); + size += nal_size; + + *dst = ptr; + *dst_size = size; + return 0; +} + static int h264_mp4toannexb_init(AVBSFContext *ctx) { H264BSFContext *s = ctx->priv_data; @@ -211,6 +259,9 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *opkt) if (j) \ av_log(__VA_ARGS__) for (int j = 0; j < 2; j++) { + int sps_count = 0; + int pps_count = 0; + buf = in->data; new_idr = s->new_idr; sps_seen = s->idr_sps_seen; @@ -241,8 +292,18 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *opkt) if (unit_type == H264_NAL_SPS) { sps_seen = new_idr = 1; + if (!j) { + h264_mp4toannexb_save_ps(&s->sps, &s->sps_size, &s->sps_buf_size, + buf, nal_size, !sps_count); + sps_count++; + } } else if (unit_type == H264_NAL_PPS) { pps_seen = new_idr = 1; + if (!j) { + h264_mp4toannexb_save_ps(&s->pps, &s->pps_size, &s->pps_buf_size, + buf, nal_size, !pps_count); + pps_count++; + } /* if SPS has not been seen yet, prepend the AVCC one to PPS */ if (!sps_seen) { if (!s->sps_size) { @@ -262,9 +323,10 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *opkt) /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */ if (new_idr && unit_type == H264_NAL_IDR_SLICE && !sps_seen && !pps_seen) { - if (ctx->par_out->extradata) - count_or_copy(&out, &out_size, ctx->par_out->extradata, - ctx->par_out->extradata_size, PS_OUT_OF_BAND, j); + if (s->sps_size) + count_or_copy(&out, &out_size, s->sps, s->sps_size, PS_OUT_OF_BAND, j); + if (s->pps_size) + count_or_copy(&out, &out_size, s->pps, s->pps_size, PS_OUT_OF_BAND, j); new_idr = 0; /* if only SPS has been seen, also insert PPS */ } else if (new_idr && unit_type == H264_NAL_IDR_SLICE && sps_seen && !pps_seen) { @@ -280,7 +342,7 @@ static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *opkt) else ps = PS_NONE; count_or_copy(&out, &out_size, buf, nal_size, ps, j); - if (!new_idr && unit_type == H264_NAL_SLICE) { + if (unit_type == H264_NAL_SLICE) { new_idr = 1; sps_seen = 0; pps_seen = 0; @@ -320,6 +382,14 @@ fail: return ret; } +static void h264_mp4toannexb_close(AVBSFContext *ctx) +{ + H264BSFContext *s = ctx->priv_data; + + av_freep(&s->sps); + av_freep(&s->pps); +} + static void h264_mp4toannexb_flush(AVBSFContext *ctx) { H264BSFContext *s = ctx->priv_data; @@ -339,5 +409,6 @@ const FFBitStreamFilter ff_h264_mp4toannexb_bsf = { .priv_data_size = sizeof(H264BSFContext), .init = h264_mp4toannexb_init, .filter = h264_mp4toannexb_filter, + .close = h264_mp4toannexb_close, .flush = h264_mp4toannexb_flush, }; diff --git a/tests/fate/h264.mak b/tests/fate/h264.mak index 0ff3d6df55..d75aebb253 100644 --- a/tests/fate/h264.mak +++ b/tests/fate/h264.mak @@ -224,7 +224,9 @@ FATE_H264-$(call FRAMECRC, MOV, H264) += fate-h264-unescaped-extradata # this sample contains field-coded frames, with both fields in a single packet FATE_H264-$(call FRAMECRC, MOV, H264) += fate-h264-twofields-packet -FATE_H264-$(call DEMMUX, MOV, H264, H264_MP4TOANNEXB_BSF) += fate-h264-bsf-mp4toannexb +FATE_H264-$(call DEMMUX, MOV, H264, H264_MP4TOANNEXB_BSF) += fate-h264-bsf-mp4toannexb \ + fate-h264-bsf-mp4toannexb-2 + FATE_H264-$(call FRAMECRC, MATROSKA, H264) += fate-h264-direct-bff FATE_H264-$(call FRAMECRC, FLV, H264, SCALE_FILTER) += fate-h264-brokensps-2580 FATE_H264-$(call FRAMECRC, MXF, H264, PCM_S24LE_DECODER SCALE_FILTER ARESAMPLE_FILTER) += fate-h264-xavc-4389 @@ -426,6 +428,10 @@ fate-h264-conformance-sva_nl1_b: CMD = framecrc -i $(TARGET_SAM fate-h264-conformance-sva_nl2_e: CMD = framecrc -i $(TARGET_SAMPLES)/h264-conformance/SVA_NL2_E.264 fate-h264-bsf-mp4toannexb: CMD = md5 -i $(TARGET_SAMPLES)/h264/interlaced_crop.mp4 -c:v copy -f h264 +# First IDR is prefixed by SPS/PPS +fate-h264-bsf-mp4toannexb-2: CMD = md5 -i $(TARGET_SAMPLES)/h264/ps_prefix_first_idr.mp4 -c:v copy -f h264 +fate-h264-bsf-mp4toannexb-2: CMP = oneline +fate-h264-bsf-mp4toannexb-2: REF = 872608399a9aac7cf91a7c447f366f32 fate-h264-crop-to-container: CMD = framemd5 -i $(TARGET_SAMPLES)/h264/crop-to-container-dims-canon.mov fate-h264-direct-bff: CMD = framecrc -i $(TARGET_SAMPLES)/h264/direct-bff.mkv