From patchwork Tue Jul 16 17:11:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: FFmpeg Technical Committee X-Patchwork-Id: 50587 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:612c:40f5:b0:482:c625:d099 with SMTP id lb53csp467106vqb; Tue, 16 Jul 2024 11:15:27 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCWT1o07ogJv5pRkewSRusqjciuMxvFd+JBV4qz1WWaWiLWb+fW3M6tNSPEeTvAL2qmGRBbIgvKqX5a3tuXA9ptt2+s1D/+G0Is95g== X-Google-Smtp-Source: AGHT+IFR0z6l1Za5Wu0GW0aJ3v+sBZR0ZVXNaRLTHr2PWdDjeSAHnSVZquq/lyBhcN+yC+1UMww+ X-Received: by 2002:a05:6402:5193:b0:57c:7ed7:897a with SMTP id 4fb4d7f45d1cf-59ef03bacaemr2352635a12.27.1721153727441; Tue, 16 Jul 2024 11:15:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721153727; cv=none; d=google.com; s=arc-20160816; b=LvV2ifi4e4r1jCozpw5LV1Gck0ugfmdSemBdGChInEuyLpalXrj6acs2MhgEvsx5/z eokcMaPeHPcVqP4TPoYRY9VvCmdIG/6Y9E5ynbw0CQQjy6KTb2lInf5GvpXm5iXjgNov ARjCbMboByJF17M6Zdg71FVOWQSKZiZHQ2GegVNLCAw6JwLO541rzVM79Bi74ih4JIND l1+ux8wb6MZU6AZtAD7XG7LGTNO86JpcRnmTWBIjIpT8poRmKb72s4ZhvGeuqwUZsQOS 0U60D2PbFbgSxcGMarzEiWg54MYiPcR1Lud0U5PzqtvRbybAT7PEu64TiMOUAer47mzG kbbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=eVrbV3ttLxc/QGcX83ZwSEIATQPynkHA8N4xK59xzkA=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=E6DkT+OUbOHnItHTeQ2pLIvkT7VzT2UpAyQ/eU+PByUKQqponmUJ69CCyzua/hCppE QmrHQER9RAUFkKLOcyGdcXDIWGkWbZdyeOGeSQxUOX/viYBTlqj0McQRzULakxyIZogD Dsbng4BkQktcK8ZkyCXGsiOMsJOSIMnBo4W8LGnjA4ZWrPZAuPgQnvRkn3hvKRNDQEVN KrPB6wZVTkAZkTptUAb3cijLqbFphKH0ykjDqsLkol0JYBHTa2An8qb+N/+v/T9VjNPi B/TmxUDZ6mZKCJydV81NM6rUTokTJho5u+2Phzks0XaQPhARdFP7i0pVucg/Juz0uj1z Ottg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@khirnov.net header.s=mail header.b=CNSmhJnU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 4fb4d7f45d1cf-59b24a6c45esi3973993a12.64.2024.07.16.11.15.27; Tue, 16 Jul 2024 11:15:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@khirnov.net header.s=mail header.b=CNSmhJnU; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1FBD968DA9B; Tue, 16 Jul 2024 21:14:20 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail1.khirnov.net (quelana.khirnov.net [94.230.150.81]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8756168DA96 for ; Tue, 16 Jul 2024 21:14:04 +0300 (EEST) Authentication-Results: mail1.khirnov.net; dkim=pass (2048-bit key; unprotected) header.d=khirnov.net header.i=@khirnov.net header.a=rsa-sha256 header.s=mail header.b=CNSmhJnU; dkim-atps=neutral Received: from localhost (mail1.khirnov.net [IPv6:::1]) by mail1.khirnov.net (Postfix) with ESMTP id D233A4DFC for ; Tue, 16 Jul 2024 19:16:34 +0200 (CEST) Received: from mail1.khirnov.net ([IPv6:::1]) by localhost (mail1.khirnov.net [IPv6:::1]) (amavis, port 10024) with ESMTP id U7YloZniHbPg for ; Tue, 16 Jul 2024 19:16:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=khirnov.net; s=mail; t=1721150188; bh=vEQWWU6S1+UgnyTwqLct9jFJzGG0yBWOIs+ec+PBZEs=; h=From:To:Subject:Date:In-Reply-To:References:From; b=CNSmhJnUgm9uNZalD/UsscxA8RW1k1vtNNF/En6SaYTYP3OtXzxG2nv3ANyLryDVR 6dHXOPci2jNFBqICK4qiYG5jjeX7RtD/faCROgp+Qw7KCs//tB/wAaLFtNrACSgshY UZCXyJJpb6VGHp26OHyA3DUmAR3gXwzFKWzbvjd2H6bJO/8gibU1CHUwgKQc29Ri5g 7DZjTGcUvalAmQf75EZFxcMINEoDtukbXIzo3Ij5xFhErGP3Nofb62KIPSGjqghjFw XLcoslmFXy7cVtIDuNdKaiFko89+6kLweTKo0yBvAC7GtSBLdU31MEwudWOIKFmjFb RtH2ETi8zQ5CQ== Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail1.khirnov.net (Postfix) with ESMTPS id EB5F84E0F for ; Tue, 16 Jul 2024 19:16:28 +0200 (CEST) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id EE1E93A3614 for ; Tue, 16 Jul 2024 19:16:21 +0200 (CEST) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Tue, 16 Jul 2024 19:11:53 +0200 Message-ID: <20240716171155.31838-38-anton@khirnov.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240716171155.31838-1-anton@khirnov.net> References: <20240716171155.31838-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 38/39] lavc/hevcdec: simplify output logic X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 9f5rngDenCDy Current code is written around the "simple" decode API's limitation that a single input packet (AU/coded frame) triggers the output of at most one output frame. However the spec contains two cases where a coded frame may cause multiple frames to be output (cf. C.5.2.2.2): * start of a new sequence * overflowing sps_max_dec_pic_buffering The decoder currently contains rather convoluted logic to handle these cases: * decode/output/per-frame sequence counters, * HEVC_FRAME_FLAG_BUMPING * ff_hevc_bump_frame() * special clauses in ff_hevc_output_frame() However, with the receive_frame() API none of that is necessary, as we can just output multiple frames at once. Previously added ContainerFifo allows that to be done in a straightforward and efficient manner. --- libavcodec/hevc/hevcdec.c | 33 +++++------- libavcodec/hevc/hevcdec.h | 32 ++++-------- libavcodec/hevc/refs.c | 104 +++++++------------------------------- 3 files changed, 42 insertions(+), 127 deletions(-) diff --git a/libavcodec/hevc/hevcdec.c b/libavcodec/hevc/hevcdec.c index 5e28bfe54e..28835f48d6 100644 --- a/libavcodec/hevc/hevcdec.c +++ b/libavcodec/hevc/hevcdec.c @@ -2906,6 +2906,7 @@ static int hevc_frame_start(HEVCContext *s) const HEVCSPS *const sps = pps->sps; int pic_size_in_ctb = ((sps->width >> sps->log2_min_cb_size) + 1) * ((sps->height >> sps->log2_min_cb_size) + 1); + int new_sequence = IS_IDR(s) || IS_BLA(s) || s->last_eos; int ret; ff_refstruct_replace(&s->pps, pps); @@ -2925,7 +2926,7 @@ static int hevc_frame_start(HEVCContext *s) return pix_fmt; s->avctx->pix_fmt = pix_fmt; - s->seq_decode = (s->seq_decode + 1) & HEVC_SEQUENCE_COUNTER_MASK; + new_sequence = 1; } memset(s->horizontal_bs, 0, s->bs_width * s->bs_height); @@ -2934,11 +2935,8 @@ static int hevc_frame_start(HEVCContext *s) memset(s->is_pcm, 0, (sps->min_pu_width + 1) * (sps->min_pu_height + 1)); memset(s->tab_slice_address, -1, pic_size_in_ctb * sizeof(*s->tab_slice_address)); - if ((IS_IDR(s) || IS_BLA(s))) { - s->seq_decode = (s->seq_decode + 1) & HEVC_SEQUENCE_COUNTER_MASK; - if (IS_IDR(s)) - ff_hevc_clear_refs(s); - } + if (IS_IDR(s)) + ff_hevc_clear_refs(s); s->slice_idx = 0; s->first_nal_type = s->nal_unit_type; @@ -2962,6 +2960,12 @@ static int hevc_frame_start(HEVCContext *s) if (pps->tiles_enabled_flag) s->local_ctx[0].end_of_tiles_x = pps->column_width[0] << sps->log2_ctb_size; + if (new_sequence) { + ret = ff_hevc_output_frames(s, 0, 0, s->sh.no_output_of_prior_pics_flag); + if (ret < 0) + return ret; + } + ret = export_stream_params_from_sei(s); if (ret < 0) return ret; @@ -3014,10 +3018,8 @@ static int hevc_frame_start(HEVCContext *s) s->cur_frame->f->pict_type = 3 - s->sh.slice_type; - if (!IS_IRAP(s)) - ff_hevc_bump_frame(s); - - ret = ff_hevc_output_frame(s, 0); + ret = ff_hevc_output_frames(s, sps->temporal_layer[sps->max_sub_layers - 1].num_reorder_pics, + sps->temporal_layer[sps->max_sub_layers - 1].max_dec_pic_buffering, 0); if (ret < 0) goto fail; @@ -3262,8 +3264,6 @@ static int decode_nal_unit(HEVCContext *s, const H2645NAL *nal) break; case HEVC_NAL_EOS_NUT: case HEVC_NAL_EOB_NUT: - s->seq_decode = (s->seq_decode + 1) & HEVC_SEQUENCE_COUNTER_MASK; - break; case HEVC_NAL_AUD: case HEVC_NAL_FD_NUT: case HEVC_NAL_UNSPEC62: @@ -3420,7 +3420,7 @@ static int hevc_receive_frame(AVCodecContext *avctx, AVFrame *frame) av_packet_unref(avpkt); ret = ff_decode_get_packet(avctx, avpkt); if (ret == AVERROR_EOF) { - ret = ff_hevc_output_frame(s, 1); + ret = ff_hevc_output_frames(s, 0, 0, 0); if (ret < 0) return ret; goto do_output; @@ -3482,7 +3482,6 @@ static int hevc_ref_frame(HEVCFrame *dst, const HEVCFrame *src) dst->poc = src->poc; dst->ctb_count = src->ctb_count; dst->flags = src->flags; - dst->sequence = src->sequence; ff_refstruct_replace(&dst->hwaccel_picture_private, src->hwaccel_picture_private); @@ -3606,8 +3605,6 @@ static int hevc_update_thread_context(AVCodecContext *dst, if ((ret = set_sps(s, s0->ps.sps)) < 0) return ret; - s->seq_decode = s0->seq_decode; - s->seq_output = s0->seq_output; s->poc_tid0 = s0->poc_tid0; s->eos = s0->eos; s->no_rasl_output_flag = s0->no_rasl_output_flag; @@ -3617,10 +3614,6 @@ static int hevc_update_thread_context(AVCodecContext *dst, s->film_grain_warning_shown = s0->film_grain_warning_shown; - if (s0->eos) { - s->seq_decode = (s->seq_decode + 1) & HEVC_SEQUENCE_COUNTER_MASK; - } - ret = ff_h2645_sei_ctx_replace(&s->sei.common, &s0->sei.common); if (ret < 0) return ret; diff --git a/libavcodec/hevc/hevcdec.h b/libavcodec/hevc/hevcdec.h index f2705b8de2..4293f1bd66 100644 --- a/libavcodec/hevc/hevcdec.h +++ b/libavcodec/hevc/hevcdec.h @@ -350,10 +350,7 @@ typedef struct DBParams { #define HEVC_FRAME_FLAG_OUTPUT (1 << 0) #define HEVC_FRAME_FLAG_SHORT_REF (1 << 1) #define HEVC_FRAME_FLAG_LONG_REF (1 << 2) -#define HEVC_FRAME_FLAG_BUMPING (1 << 3) - -#define HEVC_SEQUENCE_COUNTER_MASK 0xff -#define HEVC_SEQUENCE_COUNTER_INVALID (HEVC_SEQUENCE_COUNTER_MASK + 1) +#define HEVC_FRAME_FLAG_UNAVAILABLE (1 << 3) typedef struct HEVCFrame { union { @@ -375,12 +372,6 @@ typedef struct HEVCFrame { void *hwaccel_picture_private; ///< RefStruct reference - /** - * A sequence counter, so that old frames are output first - * after a POC reset - */ - uint16_t sequence; - /** * A combination of HEVC_FRAME_FLAG_* */ @@ -519,13 +510,6 @@ typedef struct HEVCContext { uint8_t *checksum_buf; int checksum_buf_size; - /** - * Sequence counters for decoded and output frames, so that old - * frames are output first after a POC reset - */ - uint16_t seq_decode; - uint16_t seq_output; - /** The target for the common_cabac_state of the local contexts. */ HEVCCABACState cabac; @@ -634,12 +618,16 @@ static av_always_inline int ff_hevc_nal_is_nonref(enum HEVCNALUnitType type) } /** - * Find next frame in output order and put a reference to it in frame. - * @return 1 if a frame was output, 0 otherwise + * Find frames in the DPB that are ready for output and either write them to the + * output FIFO or drop their output flag, depending on the value of discard. + * + * @param max_output maximum number of output-pending frames that can be + * present in the DPB before output is triggered + * @param max_dpb maximum number of any frames that can be present in the DPB + * before output is triggered */ -int ff_hevc_output_frame(HEVCContext *s, int flush); - -void ff_hevc_bump_frame(HEVCContext *s); +int ff_hevc_output_frames(HEVCContext *s, unsigned max_output, + unsigned max_dpb, int discard); void ff_hevc_unref_frame(HEVCFrame *frame, int flags); diff --git a/libavcodec/hevc/refs.c b/libavcodec/hevc/refs.c index 44ee7c837a..50aa06b926 100644 --- a/libavcodec/hevc/refs.c +++ b/libavcodec/hevc/refs.c @@ -131,8 +131,7 @@ int ff_hevc_set_new_ref(HEVCContext *s, int poc) for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { HEVCFrame *frame = &s->DPB[i]; - if (frame->f && frame->sequence == s->seq_decode && - frame->poc == poc) { + if (frame->f && frame->poc == poc) { av_log(s->avctx, AV_LOG_ERROR, "Duplicate POC in a sequence: %d.\n", poc); return AVERROR_INVALIDDATA; @@ -152,7 +151,6 @@ int ff_hevc_set_new_ref(HEVCContext *s, int poc) ref->flags = HEVC_FRAME_FLAG_SHORT_REF; ref->poc = poc; - ref->sequence = s->seq_decode; ref->f->crop_left = s->ps.sps->output_window.left_offset; ref->f->crop_right = s->ps.sps->output_window.right_offset; ref->f->crop_top = s->ps.sps->output_window.top_offset; @@ -165,112 +163,49 @@ static void unref_missing_refs(HEVCContext *s) { for (int i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { HEVCFrame *frame = &s->DPB[i]; - if (frame->sequence == HEVC_SEQUENCE_COUNTER_INVALID) { + if (frame->flags & HEVC_FRAME_FLAG_UNAVAILABLE) { ff_hevc_unref_frame(frame, ~0); } } } -int ff_hevc_output_frame(HEVCContext *s, int flush) +int ff_hevc_output_frames(HEVCContext *s, unsigned max_output, + unsigned max_dpb, int discard) { - if (IS_IRAP(s) && s->no_rasl_output_flag == 1) { - const static int mask = HEVC_FRAME_FLAG_BUMPING | HEVC_FRAME_FLAG_OUTPUT; - for (int i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { - HEVCFrame *frame = &s->DPB[i]; - if ((frame->flags & mask) == HEVC_FRAME_FLAG_OUTPUT && - frame->sequence != s->seq_decode) { - if (s->sh.no_output_of_prior_pics_flag == 1) - ff_hevc_unref_frame(frame, HEVC_FRAME_FLAG_OUTPUT); - else - frame->flags |= HEVC_FRAME_FLAG_BUMPING; - } - } - } - do { + while (1) { + int nb_dpb = 0; int nb_output = 0; int min_poc = INT_MAX; int i, min_idx, ret; for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { HEVCFrame *frame = &s->DPB[i]; - if ((frame->flags & HEVC_FRAME_FLAG_OUTPUT) && - frame->sequence == s->seq_output) { + if (frame->flags & HEVC_FRAME_FLAG_OUTPUT) { nb_output++; if (frame->poc < min_poc || nb_output == 1) { min_poc = frame->poc; min_idx = i; } } + nb_dpb += !!frame->flags; } - /* wait for more frames before output */ - if (!flush && s->seq_output == s->seq_decode && s->ps.sps && - nb_output <= s->ps.sps->temporal_layer[s->ps.sps->max_sub_layers - 1].num_reorder_pics) - return 0; - - if (nb_output) { + if (nb_output > max_output || + (nb_output && nb_dpb > max_dpb)) { HEVCFrame *frame = &s->DPB[min_idx]; - ret = ff_container_fifo_write(s->output_fifo, + ret = discard ? 0 : + ff_container_fifo_write(s->output_fifo, frame->needs_fg ? frame->frame_grain : frame->f); - if (frame->flags & HEVC_FRAME_FLAG_BUMPING) - ff_hevc_unref_frame(frame, HEVC_FRAME_FLAG_OUTPUT | HEVC_FRAME_FLAG_BUMPING); - else - ff_hevc_unref_frame(frame, HEVC_FRAME_FLAG_OUTPUT); + ff_hevc_unref_frame(frame, HEVC_FRAME_FLAG_OUTPUT); if (ret < 0) return ret; - av_log(s->avctx, AV_LOG_DEBUG, - "Output frame with POC %d.\n", frame->poc); - return 1; + av_log(s->avctx, AV_LOG_DEBUG, "%s frame with POC %d.\n", + discard ? "Discarded" : "Output", frame->poc); + continue; } - - if (s->seq_output != s->seq_decode) - s->seq_output = (s->seq_output + 1) & HEVC_SEQUENCE_COUNTER_MASK; - else - break; - } while (1); - - return 0; -} - -void ff_hevc_bump_frame(HEVCContext *s) -{ - int dpb = 0; - int min_poc = INT_MAX; - int i; - - for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { - HEVCFrame *frame = &s->DPB[i]; - if ((frame->flags) && - frame->sequence == s->seq_output && - frame->poc != s->poc) { - dpb++; - } - } - - if (s->ps.sps && dpb >= s->ps.sps->temporal_layer[s->ps.sps->max_sub_layers - 1].max_dec_pic_buffering) { - for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { - HEVCFrame *frame = &s->DPB[i]; - if ((frame->flags) && - frame->sequence == s->seq_output && - frame->poc != s->poc) { - if (frame->flags == HEVC_FRAME_FLAG_OUTPUT && frame->poc < min_poc) { - min_poc = frame->poc; - } - } - } - - for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { - HEVCFrame *frame = &s->DPB[i]; - if (frame->flags & HEVC_FRAME_FLAG_OUTPUT && - frame->sequence == s->seq_output && - frame->poc <= min_poc) { - frame->flags |= HEVC_FRAME_FLAG_BUMPING; - } - } - - dpb--; + return 0; } } @@ -384,7 +319,7 @@ static HEVCFrame *find_ref_idx(HEVCContext *s, int poc, uint8_t use_msb) for (i = 0; i < FF_ARRAY_ELEMS(s->DPB); i++) { HEVCFrame *ref = &s->DPB[i]; - if (ref->f && ref->sequence == s->seq_decode) { + if (ref->f) { if ((ref->poc & mask) == poc && (use_msb || ref->poc != s->poc)) return ref; } @@ -427,8 +362,7 @@ static HEVCFrame *generate_missing_ref(HEVCContext *s, int poc) } frame->poc = poc; - frame->sequence = HEVC_SEQUENCE_COUNTER_INVALID; - frame->flags = 0; + frame->flags = HEVC_FRAME_FLAG_UNAVAILABLE; if (s->avctx->active_thread_type == FF_THREAD_FRAME) ff_progress_frame_report(&frame->tf, INT_MAX);