From patchwork Wed Jan 25 16:55:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 40110 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:3ca3:b0:b9:1511:ac2c with SMTP id b35csp546487pzj; Wed, 25 Jan 2023 08:57:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXuWrxGiqGYzbctQHYbIHdBx+8yKYVmwwgU+rqQoOOIYzXqT4WMEZy7+X+Jld3Hf5cOsUgx2 X-Received: by 2002:a17:907:1710:b0:86f:64bb:47eb with SMTP id le16-20020a170907171000b0086f64bb47ebmr35392116ejc.3.1674665822319; Wed, 25 Jan 2023 08:57:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674665822; cv=none; d=google.com; s=arc-20160816; b=W/62ptp/HTj9UR3sJnN5O67+hbOLIrtghcf1fJsf4i7i7Xf6UCNv5k+knmYgKWF9xv 9yLf0f4pt3l/5fIqyDV4yK0+3hwybiUX3tBxZpyxnFnsmC3+fa/CDyr+9m7/xf3cyM6G wXaMpxa+ZFrhmUrDxzSQhF9dkVF0kI//I+5lFoASTacGEgbe/ciIU71eU4IYOg94BzQK 1bTmsFb0ZzBRg3VZPM1UPA5Wbr0k1NjRrkv50mDiD19atIj+8dgqdYQl5LL34dVzTk7x 0PGJo+Ur+QhuN3L5BQ1uhfCZMFg42U0lePTGsI3Y0JnSn+PmzZ78TE6uoPoMsj3xSdUL /t/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=ZpQY0dXpLG17d0LJvJhRbEMB1obfvMwUFF3vUO2RINA=; b=hqdGOVLUSB/T6uiH7gA/TIYLcz6blxqhMyR2dx8mPuQ1QX8tnLWzs3TQ7ud2q5QXM4 kCAINAytlwdPXpC5VBM5eiQFhcugevnVLwEIZoj1x17+/IToO42D+7r2KA9j+CVX8aVG JKrV7kA6Qqbw5NaWstYqlcR+DRciVhuOw9rSTnHWgKoZeQ8ylW5qU98O7QZ4VKmFS+TX 9Mqi5Yvu6fOG+gBrIDvrf7jRTAAIsJBm4aqK0Zyn1yXN2houZEufyJbTUzqPbxddsb/l M5tDePoEq9rWfmEtnPDvIK/7lTJhBSNNMH+3bOqre7SQ0utDi6brpFS2nggPAx8963rD /bVQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id vf5-20020a170907238500b007a953d421bbsi5824692ejb.322.2023.01.25.08.57.02; Wed, 25 Jan 2023 08:57:02 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id F34D268BDA5; Wed, 25 Jan 2023 18:56:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 892EF68BD1E for ; Wed, 25 Jan 2023 18:56:15 +0200 (EET) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id 4F2C52404F5 for ; Wed, 25 Jan 2023 17:56:15 +0100 (CET) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id ULb1ASaW01ov for ; Wed, 25 Jan 2023 17:56:14 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id 1CB0B2406CA for ; Wed, 25 Jan 2023 17:56:05 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id D63733A03EF for ; Wed, 25 Jan 2023 17:55:58 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Wed, 25 Jan 2023 17:55:26 +0100 Message-Id: <20230125165537.5371-8-anton@khirnov.net> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230125165537.5371-1-anton@khirnov.net> References: <20230125165537.5371-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 08/19] lavc/nvenc: handle frame durations and AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 9q1rlhkAU87A --- libavcodec/nvenc.c | 102 ++++++++++++++++++++++++++++++++-------- libavcodec/nvenc.h | 2 +- libavcodec/nvenc_av1.c | 3 +- libavcodec/nvenc_h264.c | 3 +- libavcodec/nvenc_hevc.c | 3 +- 5 files changed, 89 insertions(+), 24 deletions(-) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index f6df7cb6ac..ab42dc9690 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -28,6 +28,7 @@ #include "av1.h" #endif +#include "libavutil/buffer.h" #include "libavutil/hwcontext_cuda.h" #include "libavutil/hwcontext.h" #include "libavutil/cuda_check.h" @@ -162,6 +163,23 @@ static int nvenc_print_error(AVCodecContext *avctx, NVENCSTATUS err, return ret; } +typedef struct FrameData { + int64_t pts; + int64_t duration; + int64_t reordered_opaque; + + void *frame_opaque; + AVBufferRef *frame_opaque_ref; +} FrameData; + +static void reorder_queue_flush(AVFifo *queue) +{ + FrameData fd; + + while (av_fifo_read(queue, &fd, 1) >= 0) + av_buffer_unref(&fd.frame_opaque_ref); +} + typedef struct GUIDTuple { const GUID guid; int flags; @@ -1748,8 +1766,8 @@ static av_cold int nvenc_setup_surfaces(AVCodecContext *avctx) if (!ctx->surfaces) return AVERROR(ENOMEM); - ctx->timestamp_list = av_fifo_alloc2(ctx->nb_surfaces, sizeof(int64_t), 0); - if (!ctx->timestamp_list) + ctx->reorder_queue = av_fifo_alloc2(ctx->nb_surfaces, sizeof(FrameData), 0); + if (!ctx->reorder_queue) return AVERROR(ENOMEM); ctx->unused_surface_queue = av_fifo_alloc2(ctx->nb_surfaces, sizeof(NvencSurface*), 0); @@ -1833,7 +1851,8 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx) p_nvenc->nvEncEncodePicture(ctx->nvencoder, ¶ms); } - av_fifo_freep2(&ctx->timestamp_list); + reorder_queue_flush(ctx->reorder_queue); + av_fifo_freep2(&ctx->reorder_queue); av_fifo_freep2(&ctx->output_surface_ready_queue); av_fifo_freep2(&ctx->output_surface_queue); av_fifo_freep2(&ctx->unused_surface_queue); @@ -2177,18 +2196,45 @@ static void nvenc_codec_specific_pic_params(AVCodecContext *avctx, } } -static inline void timestamp_queue_enqueue(AVFifo *queue, int64_t timestamp) +static void reorder_queue_enqueue(AVFifo *queue, const AVCodecContext *avctx, + const AVFrame *frame, AVBufferRef **opaque_ref) { - av_fifo_write(queue, ×tamp, 1); + FrameData fd; + + fd.pts = frame->pts; + fd.duration = frame->duration; + fd.reordered_opaque = frame->reordered_opaque; + fd.frame_opaque = frame->opaque; + fd.frame_opaque_ref = *opaque_ref; + + *opaque_ref = NULL; + + av_fifo_write(queue, &fd, 1); } -static inline int64_t timestamp_queue_dequeue(AVFifo *queue) +static int64_t reorder_queue_dequeue(AVFifo *queue, AVCodecContext *avctx, + AVPacket *pkt) { - int64_t timestamp = AV_NOPTS_VALUE; + FrameData fd; + // The following call might fail if the queue is empty. - av_fifo_read(queue, ×tamp, 1); + if (av_fifo_read(queue, &fd, 1) < 0) + return AV_NOPTS_VALUE; + + if (pkt) { + avctx->reordered_opaque = fd.reordered_opaque; + pkt->duration = fd.duration; + + if (avctx->flags & AV_CODEC_FLAG_COPY_OPAQUE) { + pkt->opaque = fd.frame_opaque; + pkt->opaque_ref = fd.frame_opaque_ref; + fd.frame_opaque_ref = NULL; + } + } + + av_buffer_unref(&fd.frame_opaque_ref); - return timestamp; + return fd.pts; } static int nvenc_set_timestamp(AVCodecContext *avctx, @@ -2196,12 +2242,14 @@ static int nvenc_set_timestamp(AVCodecContext *avctx, AVPacket *pkt) { NvencContext *ctx = avctx->priv_data; + int64_t dts; pkt->pts = params->outputTimeStamp; + dts = reorder_queue_dequeue(ctx->reorder_queue, avctx, pkt); + if (avctx->codec_descriptor->props & AV_CODEC_PROP_REORDER) { - pkt->dts = timestamp_queue_dequeue(ctx->timestamp_list); - pkt->dts -= FFMAX(ctx->encode_config.frameIntervalP - 1, 0) * FFMAX(avctx->ticks_per_frame, 1); + pkt->dts = dts - FFMAX(ctx->encode_config.frameIntervalP - 1, 0) * FFMAX(avctx->ticks_per_frame, 1); } else { pkt->dts = pkt->pts; } @@ -2298,7 +2346,7 @@ static int process_output_surface(AVCodecContext *avctx, AVPacket *pkt, NvencSur return 0; error: - timestamp_queue_dequeue(ctx->timestamp_list); + reorder_queue_dequeue(ctx->reorder_queue, avctx, NULL); error2: return res; @@ -2528,6 +2576,8 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) int sei_count = 0; int i; + AVBufferRef *opaque_ref = NULL; + NvencContext *ctx = avctx->priv_data; NvencDynLoadFunctions *dl_fn = &ctx->nvenc_dload_funcs; NV_ENCODE_API_FUNCTION_LIST *p_nvenc = &dl_fn->nvenc_funcs; @@ -2595,9 +2645,17 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) pic_params.encodePicFlags = NV_ENC_PIC_FLAG_EOS; } + // make a reference for enqueing in the reorder queue here, + // so that reorder_queue_enqueue() cannot fail + if (frame && frame->opaque_ref && avctx->flags & AV_CODEC_FLAG_COPY_OPAQUE) { + opaque_ref = av_buffer_ref(frame->opaque_ref); + if (!opaque_ref) + return AVERROR(ENOMEM); + } + res = nvenc_push_context(avctx); if (res < 0) - return res; + goto opaque_ref_fail; nv_status = p_nvenc->nvEncEncodePicture(ctx->nvencoder, &pic_params); @@ -2606,17 +2664,17 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) res = nvenc_pop_context(avctx); if (res < 0) - return res; + goto opaque_ref_fail; if (nv_status != NV_ENC_SUCCESS && - nv_status != NV_ENC_ERR_NEED_MORE_INPUT) - return nvenc_print_error(avctx, nv_status, "EncodePicture failed!"); + nv_status != NV_ENC_ERR_NEED_MORE_INPUT) { + res = nvenc_print_error(avctx, nv_status, "EncodePicture failed!"); + goto opaque_ref_fail; + } if (frame && frame->buf[0]) { av_fifo_write(ctx->output_surface_queue, &in_surf, 1); - - if (avctx->codec_descriptor->props & AV_CODEC_PROP_REORDER) - timestamp_queue_enqueue(ctx->timestamp_list, frame->pts); + reorder_queue_enqueue(ctx->reorder_queue, avctx, frame, &opaque_ref); } /* all the pending buffers are now ready for output */ @@ -2626,6 +2684,10 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) } return 0; + +opaque_ref_fail: + av_buffer_unref(&opaque_ref); + return res; } int ff_nvenc_receive_packet(AVCodecContext *avctx, AVPacket *pkt) @@ -2684,5 +2746,5 @@ av_cold void ff_nvenc_encode_flush(AVCodecContext *avctx) NvencContext *ctx = avctx->priv_data; nvenc_send_frame(avctx, NULL); - av_fifo_reset2(ctx->timestamp_list); + reorder_queue_flush(ctx->reorder_queue); } diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h index 05a7ac48b1..411c83aa94 100644 --- a/libavcodec/nvenc.h +++ b/libavcodec/nvenc.h @@ -171,7 +171,7 @@ typedef struct NvencContext AVFifo *unused_surface_queue; AVFifo *output_surface_queue; AVFifo *output_surface_ready_queue; - AVFifo *timestamp_list; + AVFifo *reorder_queue; NV_ENC_SEI_PAYLOAD *sei_data; int sei_data_size; diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c index 2ed99d948b..2b349c7b61 100644 --- a/libavcodec/nvenc_av1.c +++ b/libavcodec/nvenc_av1.c @@ -181,7 +181,8 @@ const FFCodec ff_av1_nvenc_encoder = { .defaults = defaults, .p.pix_fmts = ff_nvenc_pix_fmts, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.wrapper_name = "nvenc", diff --git a/libavcodec/nvenc_h264.c b/libavcodec/nvenc_h264.c index a69358b03b..5dc2961c3b 100644 --- a/libavcodec/nvenc_h264.c +++ b/libavcodec/nvenc_h264.c @@ -232,7 +232,8 @@ const FFCodec ff_h264_nvenc_encoder = { .p.priv_class = &h264_nvenc_class, .defaults = defaults, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.pix_fmts = ff_nvenc_pix_fmts, diff --git a/libavcodec/nvenc_hevc.c b/libavcodec/nvenc_hevc.c index 5ad423444a..1362a927c8 100644 --- a/libavcodec/nvenc_hevc.c +++ b/libavcodec/nvenc_hevc.c @@ -214,7 +214,8 @@ const FFCodec ff_hevc_nvenc_encoder = { .defaults = defaults, .p.pix_fmts = ff_nvenc_pix_fmts, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.wrapper_name = "nvenc",