From patchwork Sun Jan 21 14:18:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianfeng Zheng X-Patchwork-Id: 45689 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:a402:b0:199:de12:6fa6 with SMTP id z2csp943800pzk; Sun, 21 Jan 2024 06:19:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5BWvAIrC7ggpa4lN4nhsZ73abvdyA7Z/jdGfAFe54sOjiGUfIBHxgYDYt1kO7QskKRyba X-Received: by 2002:a05:600c:3ba2:b0:40e:6707:b758 with SMTP id n34-20020a05600c3ba200b0040e6707b758mr1424508wms.178.1705846781384; Sun, 21 Jan 2024 06:19:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705846781; cv=none; d=google.com; s=arc-20160816; b=SPNAcN9UCHMB0nH2xgvxsASNu4h0nHvlmvluVZVAd6yheP/z2uIKAYn5Efr5zulfTI UJRI3MoxfwvNqKvCDCa3TJVl3ncUk0FFMfoHIw4iCCQJsm9f1AAPPy2+rg7MhOKelM+8 pAIeyIzwbLcqZIAV6qoieKDyC9tPX6jXYyEEdWgXXl6NbzD6pFztXnS1oLmJjEmORSlf lyddStxCHvT0ZFVDAvCFKzUGmVvhALwvMW+mE1ipX/btCHF/O8SzNnIC/hh10X34aAo8 I6rXzJyjOe5pbjL95p1aJ2x3IXNK5kP0xJMaNeRQKXTvdmo7PrnQe7FhHWfaoa/dlRTx 3C6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to; bh=ot5JykD3caer4UPbkwHqCGUrB+XcQjX50cFRNYbl96g=; fh=iiAW4vjlCGdXV3Yszi/eNCPr6t9/oDQbs1mdNL59KvI=; b=m5Le8L9kKSKC/498XGLdS44EGbGufSENdKt5aoXlgzKyFb48PhXceZI13NretH/y+l 0p4xhzxL+B0cedwALCJuksRwXT3Bl2r/aWqBvSj/zeQg+weDH1qhId9OALfGGlB5NjrO Fx1moXMRTKqJDswPOYLTN/ESvjp/ELTx6Xp9HLcBdsqDX/cfvtQ420eeEXGNj+mLGlU3 VN4FIf94NFzFRauC4cTuNrPlMQWGtm9erV9I6XiaxPmCkQMI9m3ht+JI4slmq8uK77RH 3LIJ3zprCCKAwJeb18YLIgzMkis/j1KBpHo9TlJi2NQZULZ4PHHVX6YQRaJoeKjKhJp+ Wr+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=I5nggDvV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id q4-20020a056402248400b0055872d16f9asi10681380eda.116.2024.01.21.06.19.40; Sun, 21 Jan 2024 06:19:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=I5nggDvV; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 4375368CFAA; Sun, 21 Jan 2024 16:19:23 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2086D68CBFD for ; Sun, 21 Jan 2024 16:19:17 +0200 (EET) Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1d70696b6faso20636235ad.3 for ; Sun, 21 Jan 2024 06:19:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705846755; x=1706451555; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KNBkGSne5WCsI0mJWlh0m6IY59bx9aN219RNZMYrzX0=; b=I5nggDvVgH0NopaHu2/k9cUaZ59t6gM1yVgdqs81WKv3eknZz7h8eXQasGdCU+LCQp 7UaNSIJr9040YK87HxJkKJHLl5yLt0ZpauPkVJzmk8rThQDEjUk83pivp0N7kKDS3iFm VHH2qC4iCrS0OQNbKtatai8R+NkbpVRRMk/tdWWESrzlM0KK7qj2KqB/u/p5NaMy3fUI KUR+zO18ngxAKs/R8DzpH3Jcrciq0LjZjDj/WS3R/7QycxrVXUNQ9r4j4jfGo5k6uQqC fwAcUT/wjAcYH2FFUFUboC7BACk7BESfswmDGyztREF6aeqJ8QoV+S3TuemVxIftd5zq fQCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705846755; x=1706451555; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KNBkGSne5WCsI0mJWlh0m6IY59bx9aN219RNZMYrzX0=; b=JNxvemeNAaaGSVvAtSJd0nqlRzxmbzvv6jeOQ9k1iyUkh6ym7/3bWLEMVeRqU/mkW/ DSl29wC4CwR87tCWDVhT1BQKA5QVwftbxk7z5kVC0OtOTUPnekJnggGjJsRg7xEivXlO z5n39qzB51vGJqH0H65g0D6+gg2zOQk+K3/xyk6YdWnzXBOxdhbbJynWDPsSUoQhj5CX pKIKNrUPAF8kZMJi4dtqHkK4LZAltnsci/ea+cNsQErpIDbn8zb4kGWCy+ZDLGSYZbOW Wu88EfWG3zxdTSirJjuGICK7ctwfqz/e5tCFajIxwWXVH1n27dl3CPEN07TTnezfF9Rt b2tQ== X-Gm-Message-State: AOJu0Yz6UxHQIhta8ZnrlpIU5LP439hQzuu2mXW+INJAPMJ4iHC9JFiE UBSYG7tGccqSn0aRoEhtsbpsxl/7y/D//csIymSWMJjoMy86YJHgOdkAqv8zw8s= X-Received: by 2002:a17:902:7046:b0:1d4:dddc:d38e with SMTP id h6-20020a170902704600b001d4dddcd38emr2982811plt.61.1705846754705; Sun, 21 Jan 2024 06:19:14 -0800 (PST) Received: from localhost.localdomain ([183.242.63.6]) by smtp.gmail.com with ESMTPSA id bg4-20020a1709028e8400b001d4b73f60e4sm5840726plb.149.2024.01.21.06.19.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Jan 2024 06:19:14 -0800 (PST) From: "jianfeng.zheng" X-Google-Original-From: "jianfeng.zheng" To: ffmpeg-devel@ffmpeg.org Date: Sun, 21 Jan 2024 22:18:43 +0800 Message-Id: <20240121141846.4077778-4-jianfeng.zheng@mthreads.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240121141846.4077778-1-jianfeng.zheng@mthreads.com> References: <20240121141846.4077778-1-jianfeng.zheng@mthreads.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [vaapi-cavs 4/7] cavs: fix dpb reorder issues when 'low_delay' is varied X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: "jianfeng.zheng" Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 1dSuBiMSz12X Consider multi sequences in one stream, 'low_delay' may change between sequences. Signed-off-by: jianfeng.zheng --- libavcodec/cavs.c | 12 +++++ libavcodec/cavs.h | 2 + libavcodec/cavsdec.c | 105 +++++++++++++++++++++++++++++++++---------- 3 files changed, 95 insertions(+), 24 deletions(-) diff --git a/libavcodec/cavs.c b/libavcodec/cavs.c index fdd577f7fb..ed7b278336 100644 --- a/libavcodec/cavs.c +++ b/libavcodec/cavs.c @@ -810,6 +810,14 @@ av_cold int ff_cavs_init(AVCodecContext *avctx) if (!h->cur.f || !h->DPB[0].f || !h->DPB[1].f) return AVERROR(ENOMEM); + h->out[0].f = av_frame_alloc(); + h->out[1].f = av_frame_alloc(); + h->out[2].f = av_frame_alloc(); + if (!h->out[0].f || !h->out[1].f || !h->out[2].f) { + ff_cavs_end(avctx); + return AVERROR(ENOMEM); + } + h->luma_scan[0] = 0; h->luma_scan[1] = 8; h->intra_pred_l[INTRA_L_VERT] = intra_pred_vert; @@ -840,6 +848,10 @@ av_cold int ff_cavs_end(AVCodecContext *avctx) av_frame_free(&h->DPB[0].f); av_frame_free(&h->DPB[1].f); + av_frame_free(&h->out[0].f); + av_frame_free(&h->out[1].f); + av_frame_free(&h->out[2].f); + av_freep(&h->top_qp); av_freep(&h->top_mv[0]); av_freep(&h->top_mv[1]); diff --git a/libavcodec/cavs.h b/libavcodec/cavs.h index ad49abff92..f490657959 100644 --- a/libavcodec/cavs.h +++ b/libavcodec/cavs.h @@ -166,6 +166,7 @@ struct dec_2dvlc { typedef struct AVSFrame { AVFrame *f; int poc; + int outputed; } AVSFrame; typedef struct AVSContext { @@ -177,6 +178,7 @@ typedef struct AVSContext { GetBitContext gb; AVSFrame cur; ///< currently decoded frame AVSFrame DPB[2]; ///< reference frames + AVSFrame out[3]; ///< output queue, size 2 maybe enough int dist[2]; ///< temporal distances from current frame to ref frames int low_delay; int profile, level; diff --git a/libavcodec/cavsdec.c b/libavcodec/cavsdec.c index 9ad0f29b01..6f462d861c 100644 --- a/libavcodec/cavsdec.c +++ b/libavcodec/cavsdec.c @@ -1056,6 +1056,44 @@ static inline int check_for_slice(AVSContext *h) * ****************************************************************************/ +/** + * @brief remove frame out of dpb + */ +static void cavs_frame_unref(AVSFrame *frame) +{ + /* frame->f can be NULL if context init failed */ + if (!frame->f || !frame->f->buf[0]) + return; + + av_frame_unref(frame->f); +} + +static int output_one_frame(AVSContext *h, AVFrame *data, int *got_frame) +{ + if (h->out[0].f->buf[0]) { + av_log(h->avctx, AV_LOG_DEBUG, "output frame: poc=%d\n", h->out[0].poc); + av_frame_move_ref(data, h->out[0].f); + *got_frame = 1; + + // out[0] <- out[1] <- out[2] <- out[0] + cavs_frame_unref(&h->out[2]); + FFSWAP(AVSFrame, h->out[0], h->out[2]); + FFSWAP(AVSFrame, h->out[0], h->out[1]); + + return 1; + } + + return 0; +} + +static void queue_one_frame(AVSContext *h, AVSFrame *out) +{ + int idx = !h->out[0].f->buf[0] ? 0 : (!h->out[1].f->buf[0] ? 1 : 2); + av_log(h->avctx, AV_LOG_DEBUG, "queue in out[%d]: poc=%d\n", idx, out->poc); + av_frame_ref(h->out[idx].f, out->f); + h->out[idx].poc = out->poc; +} + static int decode_pic(AVSContext *h) { int ret; @@ -1068,7 +1106,7 @@ static int decode_pic(AVSContext *h) return AVERROR_INVALIDDATA; } - av_frame_unref(h->cur.f); + cavs_frame_unref(&h->cur); skip_bits(&h->gb, 16);//bbv_dwlay if (h->stc == PIC_PB_START_CODE) { @@ -1077,10 +1115,13 @@ static int decode_pic(AVSContext *h) av_log(h->avctx, AV_LOG_ERROR, "illegal picture type\n"); return AVERROR_INVALIDDATA; } + /* make sure we have the reference frames we need */ - if (!h->DPB[0].f->data[0] || - (!h->DPB[1].f->data[0] && h->cur.f->pict_type == AV_PICTURE_TYPE_B)) + if (!h->DPB[0].f->buf[0] || + (!h->DPB[1].f->buf[0] && h->cur.f->pict_type == AV_PICTURE_TYPE_B)) { + av_log(h->avctx, AV_LOG_ERROR, "Invalid reference frame\n"); return AVERROR_INVALIDDATA; + } } else { h->cur.f->pict_type = AV_PICTURE_TYPE_I; if (get_bits1(&h->gb)) { //time_code @@ -1124,6 +1165,8 @@ static int decode_pic(AVSContext *h) if ((ret = ff_cavs_init_pic(h)) < 0) return ret; h->cur.poc = get_bits(&h->gb, 8) * 2; + av_log(h->avctx, AV_LOG_DEBUG, "poc=%d, type=%d\n", + h->cur.poc, h->cur.f->pict_type); /* get temporal distances and MV scaling factors */ if (h->cur.f->pict_type != AV_PICTURE_TYPE_B) { @@ -1137,6 +1180,8 @@ static int decode_pic(AVSContext *h) if (h->cur.f->pict_type == AV_PICTURE_TYPE_B) { h->sym_factor = h->dist[0] * h->scale_den[1]; if (FFABS(h->sym_factor) > 32768) { + av_log(h->avctx, AV_LOG_ERROR, "poc=%d/%d/%d, dist=%d/%d\n", + h->DPB[1].poc, h->DPB[0].poc, h->cur.poc, h->dist[0], h->dist[1]); av_log(h->avctx, AV_LOG_ERROR, "sym_factor %d too large\n", h->sym_factor); return AVERROR_INVALIDDATA; } @@ -1250,11 +1295,6 @@ static int decode_pic(AVSContext *h) } while (ff_cavs_next_mb(h)); } emms_c(); - if (ret >= 0 && h->cur.f->pict_type != AV_PICTURE_TYPE_B) { - av_frame_unref(h->DPB[1].f); - FFSWAP(AVSFrame, h->cur, h->DPB[1]); - FFSWAP(AVSFrame, h->DPB[0], h->DPB[1]); - } return ret; } @@ -1337,11 +1377,12 @@ static int cavs_decode_frame(AVCodecContext *avctx, AVFrame *rframe, const uint8_t *buf_ptr; int frame_start = 0; - if (buf_size == 0) { - if (!h->low_delay && h->DPB[0].f->data[0]) { - *got_frame = 1; - av_frame_move_ref(rframe, h->DPB[0].f); + if (avpkt->size == 0) { + if (h->DPB[0].f->buf[0] && !h->DPB[0].outputed) { + queue_one_frame(h, &h->DPB[0]); + cavs_frame_unref(&h->DPB[0]); } + output_one_frame(h, rframe, got_frame); return 0; } @@ -1364,8 +1405,8 @@ static int cavs_decode_frame(AVCodecContext *avctx, AVFrame *rframe, break; case PIC_I_START_CODE: if (!h->got_keyframe) { - av_frame_unref(h->DPB[0].f); - av_frame_unref(h->DPB[1].f); + cavs_frame_unref(&h->DPB[0]); + cavs_frame_unref(&h->DPB[1]); h->got_keyframe = 1; } case PIC_PB_START_CODE: @@ -1375,23 +1416,39 @@ static int cavs_decode_frame(AVCodecContext *avctx, AVFrame *rframe, if (*got_frame) av_frame_unref(rframe); *got_frame = 0; - if (!h->got_keyframe) + if (!h->got_keyframe) { + av_log(avctx, AV_LOG_ERROR, "No keyframe decoded before P/B frame.\n"); break; + } init_get_bits(&h->gb, buf_ptr, input_size); h->stc = stc; - if (decode_pic(h)) - break; - *got_frame = 1; + if ((ret = decode_pic(h)) < 0) + return ret; + buf_ptr = align_get_bits(&h->gb); + + h->cur.outputed = 0; if (h->cur.f->pict_type != AV_PICTURE_TYPE_B) { - if (h->DPB[!h->low_delay].f->data[0]) { - if ((ret = av_frame_ref(rframe, h->DPB[!h->low_delay].f)) < 0) - return ret; - } else { - *got_frame = 0; + // at most one delay + if (h->DPB[0].f->buf[0] && !h->DPB[0].outputed) { + queue_one_frame(h, &h->DPB[0]); + h->DPB[0].outputed = 1; } + + if (h->low_delay) { + queue_one_frame(h, &h->cur); + h->cur.outputed = 1; + } + + // null -> curr -> DPB[0] -> DPB[1] + cavs_frame_unref(&h->DPB[1]); + FFSWAP(AVSFrame, h->cur, h->DPB[1]); + FFSWAP(AVSFrame, h->DPB[0], h->DPB[1]); } else { - av_frame_move_ref(rframe, h->cur.f); + queue_one_frame(h, &h->cur); + cavs_frame_unref(&h->cur); } + + output_one_frame(h, rframe, got_frame); break; case EXT_START_CODE: //mpeg_decode_extension(avctx, buf_ptr, input_size);