From patchwork Sun Nov 27 17:03:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Khirnov X-Patchwork-Id: 39571 Delivered-To: andriy.gelman@gmail.com Received: by 2002:a05:6a11:561f:b0:33d:fc04:f431 with SMTP id rr31csp8295285pxb; Sun, 27 Nov 2022 09:08:58 -0800 (PST) X-Google-Smtp-Source: AA0mqf73I4ry7cvR6tKMWRN2h0Q3lUjHO7I5cyq+5ss0kQfiklZeA1+V+faOWAZpTcyH9qa0hEVX X-Received: by 2002:a17:907:d60c:b0:7ae:e592:6d83 with SMTP id wd12-20020a170907d60c00b007aee5926d83mr16825183ejc.699.1669568937982; Sun, 27 Nov 2022 09:08:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669568937; cv=none; d=google.com; s=arc-20160816; b=UmNnRUnlYw/0Ywr1cg6hu9xDH069wHPJv9iPzf2dCJT/8luWh3XDQfoyAfQfsftNUQ 2bBnm1YLW7scLHIT+ZlAngJdLbWkc/hhE9mxzZqatr9bON7WAABJOShtOmovQ4QlY6i2 Wp5WCbLWhaE7+ZOtKx3S3qwOlRQeEVK3HD3T1b2KmEEqXn2tgGPq1gpTt+os/bHwJg6r 56oaTbzsrSXSYllfpED+ZxHRtzNrvP5t0jU4SzJn/Wwg5kAcd7tZexNl1BcVvNQKDaw3 3YhyxrY2iQthP7iNXEoqz8jdIX1OAy/86NlnDhdbCzGWa0gtGLRtGR1G+Te3MxJMRR8p tmHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:delivered-to; bh=jawVmwB2UD4vCYePZgp6jnePhwUTz7cFRcQzVsVD3X8=; b=yJ38EoKC+O4ho48wkpPPePz5Z9X1eVQDlKkwU9oADvqKcd0WEI0wOgSnJXhajPTNWl PNrSgDMIl0ML6Yad1ynih49vNIAPdGKV31fsEdtAnhWxOZnsh5lk6RNSLEPV902d4l32 CfoDH5RtXhwG+Ic3i5qFMdzrgtYzWP8QXIwBZqqu6OXTKZM79iGL0OFL7rStMdWSGrvq V+rUd+tWtUEARTBQLY/Je3AMt79KLq+dpBhGCm7egdTILH6IJ8fKoY9vch+wTKUpy+Tk hnQRpYBXN7demgKZRwYhOoXfdk8pwbYznC46CejachEK+NYY8c9dyRI7JrAi8XS6IZmv k9gg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ds12-20020a170907724c00b007bc93454b3fsi7285091ejc.953.2022.11.27.09.08.57; Sun, 27 Nov 2022 09:08:57 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0D13468BC2A; Sun, 27 Nov 2022 19:06:19 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail0.khirnov.net (red.khirnov.net [176.97.15.12]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CEA9768B735 for ; Sun, 27 Nov 2022 19:05:58 +0200 (EET) Received: from localhost (localhost [IPv6:::1]) by mail0.khirnov.net (Postfix) with ESMTP id D9FBB2406CA for ; Sun, 27 Nov 2022 18:05:55 +0100 (CET) Received: from mail0.khirnov.net ([IPv6:::1]) by localhost (mail0.khirnov.net [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id X0dMjoxAYmmK for ; Sun, 27 Nov 2022 18:05:55 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:2a00:c500:561:201::7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "libav.khirnov.net", Issuer "smtp.khirnov.net SMTP CA" (verified OK)) by mail0.khirnov.net (Postfix) with ESMTPS id B9E4B2406CC for ; Sun, 27 Nov 2022 18:05:51 +0100 (CET) Received: from libav.khirnov.net (libav.khirnov.net [IPv6:::1]) by libav.khirnov.net (Postfix) with ESMTP id B115B3A25EC for ; Sun, 27 Nov 2022 18:05:46 +0100 (CET) From: Anton Khirnov To: ffmpeg-devel@ffmpeg.org Date: Sun, 27 Nov 2022 18:03:41 +0100 Message-Id: <20221127170351.11477-20-anton@khirnov.net> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221127170351.11477-1-anton@khirnov.net> References: <20221127170351.11477-1-anton@khirnov.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 20/30] lavc/nvenc: handle frame durations and AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: XF/xyrlc0OfW Content-Length: 9931 --- libavcodec/nvenc.c | 102 ++++++++++++++++++++++++++++++++-------- libavcodec/nvenc.h | 2 +- libavcodec/nvenc_av1.c | 3 +- libavcodec/nvenc_h264.c | 3 +- libavcodec/nvenc_hevc.c | 3 +- 5 files changed, 89 insertions(+), 24 deletions(-) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index 9726c565d3..80f78155f2 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -28,6 +28,7 @@ #include "av1.h" #endif +#include "libavutil/buffer.h" #include "libavutil/hwcontext_cuda.h" #include "libavutil/hwcontext.h" #include "libavutil/cuda_check.h" @@ -162,6 +163,23 @@ static int nvenc_print_error(AVCodecContext *avctx, NVENCSTATUS err, return ret; } +typedef struct FrameData { + int64_t pts; + int64_t duration; + int64_t reordered_opaque; + + void *frame_opaque; + AVBufferRef *frame_opaque_ref; +} FrameData; + +static void reorder_queue_flush(AVFifo *queue) +{ + FrameData fd; + + while (av_fifo_read(queue, &fd, 1) >= 0) + av_buffer_unref(&fd.frame_opaque_ref); +} + typedef struct GUIDTuple { const GUID guid; int flags; @@ -1743,8 +1761,8 @@ static av_cold int nvenc_setup_surfaces(AVCodecContext *avctx) if (!ctx->surfaces) return AVERROR(ENOMEM); - ctx->timestamp_list = av_fifo_alloc2(ctx->nb_surfaces, sizeof(int64_t), 0); - if (!ctx->timestamp_list) + ctx->reorder_queue = av_fifo_alloc2(ctx->nb_surfaces, sizeof(FrameData), 0); + if (!ctx->reorder_queue) return AVERROR(ENOMEM); ctx->unused_surface_queue = av_fifo_alloc2(ctx->nb_surfaces, sizeof(NvencSurface*), 0); @@ -1828,7 +1846,8 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx) p_nvenc->nvEncEncodePicture(ctx->nvencoder, ¶ms); } - av_fifo_freep2(&ctx->timestamp_list); + reorder_queue_flush(ctx->reorder_queue); + av_fifo_freep2(&ctx->reorder_queue); av_fifo_freep2(&ctx->output_surface_ready_queue); av_fifo_freep2(&ctx->output_surface_queue); av_fifo_freep2(&ctx->unused_surface_queue); @@ -2172,18 +2191,45 @@ static void nvenc_codec_specific_pic_params(AVCodecContext *avctx, } } -static inline void timestamp_queue_enqueue(AVFifo *queue, int64_t timestamp) +static void reorder_queue_enqueue(AVFifo *queue, const AVCodecContext *avctx, + const AVFrame *frame, AVBufferRef **opaque_ref) { - av_fifo_write(queue, ×tamp, 1); + FrameData fd; + + fd.pts = frame->pts; + fd.duration = frame->duration; + fd.reordered_opaque = frame->reordered_opaque; + fd.frame_opaque = frame->opaque; + fd.frame_opaque_ref = *opaque_ref; + + *opaque_ref = NULL; + + av_fifo_write(queue, &fd, 1); } -static inline int64_t timestamp_queue_dequeue(AVFifo *queue) +static int64_t reorder_queue_dequeue(AVFifo *queue, AVCodecContext *avctx, + AVPacket *pkt) { - int64_t timestamp = AV_NOPTS_VALUE; + FrameData fd; + // The following call might fail if the queue is empty. - av_fifo_read(queue, ×tamp, 1); + if (av_fifo_read(queue, &fd, 1) < 0) + return AV_NOPTS_VALUE; + + if (pkt) { + avctx->reordered_opaque = fd.reordered_opaque; + pkt->duration = fd.duration; + + if (avctx->flags & AV_CODEC_FLAG_COPY_OPAQUE) { + pkt->opaque = fd.frame_opaque; + pkt->opaque_ref = fd.frame_opaque_ref; + fd.frame_opaque_ref = NULL; + } + } + + av_buffer_unref(&fd.frame_opaque_ref); - return timestamp; + return fd.pts; } static int nvenc_set_timestamp(AVCodecContext *avctx, @@ -2191,12 +2237,14 @@ static int nvenc_set_timestamp(AVCodecContext *avctx, AVPacket *pkt) { NvencContext *ctx = avctx->priv_data; + int64_t dts; pkt->pts = params->outputTimeStamp; + dts = reorder_queue_dequeue(ctx->reorder_queue, avctx, pkt); + if (avctx->codec_descriptor->props & AV_CODEC_PROP_REORDER) { - pkt->dts = timestamp_queue_dequeue(ctx->timestamp_list); - pkt->dts -= FFMAX(ctx->encode_config.frameIntervalP - 1, 0) * FFMAX(avctx->ticks_per_frame, 1); + pkt->dts = dts - FFMAX(ctx->encode_config.frameIntervalP - 1, 0) * FFMAX(avctx->ticks_per_frame, 1); } else { pkt->dts = pkt->pts; } @@ -2293,7 +2341,7 @@ static int process_output_surface(AVCodecContext *avctx, AVPacket *pkt, NvencSur return 0; error: - timestamp_queue_dequeue(ctx->timestamp_list); + reorder_queue_dequeue(ctx->reorder_queue, avctx, NULL); error2: return res; @@ -2523,6 +2571,8 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) int sei_count = 0; int i; + AVBufferRef *opaque_ref = NULL; + NvencContext *ctx = avctx->priv_data; NvencDynLoadFunctions *dl_fn = &ctx->nvenc_dload_funcs; NV_ENCODE_API_FUNCTION_LIST *p_nvenc = &dl_fn->nvenc_funcs; @@ -2590,9 +2640,17 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) pic_params.encodePicFlags = NV_ENC_PIC_FLAG_EOS; } + // make a reference for enqueing in the reorder queue here, + // so that reorder_queue_enqueue() cannot fail + if (frame && frame->opaque_ref && avctx->flags & AV_CODEC_FLAG_COPY_OPAQUE) { + opaque_ref = av_buffer_ref(frame->opaque_ref); + if (!opaque_ref) + return AVERROR(ENOMEM); + } + res = nvenc_push_context(avctx); if (res < 0) - return res; + goto opaque_ref_fail; nv_status = p_nvenc->nvEncEncodePicture(ctx->nvencoder, &pic_params); @@ -2601,17 +2659,17 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) res = nvenc_pop_context(avctx); if (res < 0) - return res; + goto opaque_ref_fail; if (nv_status != NV_ENC_SUCCESS && - nv_status != NV_ENC_ERR_NEED_MORE_INPUT) - return nvenc_print_error(avctx, nv_status, "EncodePicture failed!"); + nv_status != NV_ENC_ERR_NEED_MORE_INPUT) { + res = nvenc_print_error(avctx, nv_status, "EncodePicture failed!"); + goto opaque_ref_fail; + } if (frame && frame->buf[0]) { av_fifo_write(ctx->output_surface_queue, &in_surf, 1); - - if (avctx->codec_descriptor->props & AV_CODEC_PROP_REORDER) - timestamp_queue_enqueue(ctx->timestamp_list, frame->pts); + reorder_queue_enqueue(ctx->reorder_queue, avctx, frame, &opaque_ref); } /* all the pending buffers are now ready for output */ @@ -2621,6 +2679,10 @@ static int nvenc_send_frame(AVCodecContext *avctx, const AVFrame *frame) } return 0; + +opaque_ref_fail: + av_buffer_unref(&opaque_ref); + return res; } int ff_nvenc_receive_packet(AVCodecContext *avctx, AVPacket *pkt) @@ -2679,5 +2741,5 @@ av_cold void ff_nvenc_encode_flush(AVCodecContext *avctx) NvencContext *ctx = avctx->priv_data; nvenc_send_frame(avctx, NULL); - av_fifo_reset2(ctx->timestamp_list); + reorder_queue_flush(ctx->reorder_queue); } diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h index 05a7ac48b1..411c83aa94 100644 --- a/libavcodec/nvenc.h +++ b/libavcodec/nvenc.h @@ -171,7 +171,7 @@ typedef struct NvencContext AVFifo *unused_surface_queue; AVFifo *output_surface_queue; AVFifo *output_surface_ready_queue; - AVFifo *timestamp_list; + AVFifo *reorder_queue; NV_ENC_SEI_PAYLOAD *sei_data; int sei_data_size; diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c index 2ed99d948b..2b349c7b61 100644 --- a/libavcodec/nvenc_av1.c +++ b/libavcodec/nvenc_av1.c @@ -181,7 +181,8 @@ const FFCodec ff_av1_nvenc_encoder = { .defaults = defaults, .p.pix_fmts = ff_nvenc_pix_fmts, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.wrapper_name = "nvenc", diff --git a/libavcodec/nvenc_h264.c b/libavcodec/nvenc_h264.c index a69358b03b..5dc2961c3b 100644 --- a/libavcodec/nvenc_h264.c +++ b/libavcodec/nvenc_h264.c @@ -232,7 +232,8 @@ const FFCodec ff_h264_nvenc_encoder = { .p.priv_class = &h264_nvenc_class, .defaults = defaults, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.pix_fmts = ff_nvenc_pix_fmts, diff --git a/libavcodec/nvenc_hevc.c b/libavcodec/nvenc_hevc.c index 5ad423444a..1362a927c8 100644 --- a/libavcodec/nvenc_hevc.c +++ b/libavcodec/nvenc_hevc.c @@ -214,7 +214,8 @@ const FFCodec ff_hevc_nvenc_encoder = { .defaults = defaults, .p.pix_fmts = ff_nvenc_pix_fmts, .p.capabilities = AV_CODEC_CAP_DELAY | AV_CODEC_CAP_HARDWARE | - AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1, + AV_CODEC_CAP_ENCODER_FLUSH | AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, .caps_internal = FF_CODEC_CAP_NOT_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP, .p.wrapper_name = "nvenc",