From patchwork Thu Sep 19 23:16:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Carlos Ruiz X-Patchwork-Id: 51662 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:d154:0:b0:48e:c0f8:d0de with SMTP id bt20csp628305vqb; Thu, 19 Sep 2024 16:16:40 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVgOqouTCSXPf1LzqyO+qz/hVeUzG1SME9vA8fkY3CltlCI24YufzxJtUieGOBpFLxWF+PlazLY7uLia3M2WAfc@gmail.com X-Google-Smtp-Source: AGHT+IGkMHQli9sSsl9xfTjeRiHSKWHwgOLSMdRxyHHS87tf/42R2CDvLhN62qUd/l1MML7CsHo7 X-Received: by 2002:a17:907:6d26:b0:a8d:2623:dd19 with SMTP id a640c23a62f3a-a90d510f250mr19600566b.14.1726787800407; Thu, 19 Sep 2024 16:16:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1726787800; cv=none; d=google.com; s=arc-20240605; b=hVmKRsTrrzmVNYC/vgCEUwWHpGFqqI3/nzLKLChCFUMsNbnLoPZ7S95gDhns8qcQzA Or1C0qomtx00WciSkwZSDFagx7XMeXT/sfdUN3BkZDqb3RW7KNJGEvRWC2RKNCQBun5/ 81Miv1UP4B8QHlMswzEDW8J1N0SCT9p7Qc7/p92mQ/IQdHEHmRUv6rJBKhhBEfYoXCDu r3qZtMBstUIEWjMAVAOGjJjG4NsyzdyvWqC0ffiiNX4p6BNNpuV/S8KCUs8J5bq8JA7Z idnFLYKsi8PES/ughnkZ4hjU1RUOR/9uyztnYWxa++T4YH5dHCzOAMkluo5JRebqEHQe u47g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:to:message-id:date:from:mime-version :dkim-signature:delivered-to; bh=sswiZ2a71E00WeJ9p20ATnkB8UhtX7dKrfOobVhhh3s=; fh=YOA8vD9MJZuwZ71F/05pj6KdCjf6jQRmzLS+CATXUQk=; b=KMHZVMRoJZJnW7dCS4y0GhUEnsB3VtdEBHjMXXWKDKVENjN4l66NHVAUmR1sEnxxdt VMerGJ+t5SdIyq/1kuPeu/RJeWmbEVHIoPKn45B6MqcunP4+thCs7gdeJD/yuBH5UPOv 2PzMC+nWF5Vgo9fAiz0dt5JsYaM3K6akvG9eg/gNJEgnPndQjVoi0wob3xLoFrOPk/bl pBfTHmM5s3aVSOTmcJTkShRvbIXtNmqrD6E57KiGygeqUTFYyNerhGx1CaZGEXibp1OI x81M9PQsxW6d07Ho7avAcYZwagLvEPOCmIJhWUQ6gOfFX56ACyLu9tzVdjToP4x4bVdY Js/w==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=neJMj8ms; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id a640c23a62f3a-a90612ef655si854323866b.508.2024.09.19.16.16.40; Thu, 19 Sep 2024 16:16:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20230601 header.b=neJMj8ms; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com; dara=fail header.i=@gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2832F68D9F8; Fri, 20 Sep 2024 02:16:37 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E272168D9F8 for ; Fri, 20 Sep 2024 02:16:29 +0300 (EEST) Received: by mail-io1-f41.google.com with SMTP id ca18e2360f4ac-82a109bc459so92579039f.1 for ; Thu, 19 Sep 2024 16:16:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726787788; x=1727392588; darn=ffmpeg.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=KVQBjSrLfeVCjZ7sTnJ78+yxGiIQHaOkWqkjn6pjxFQ=; b=neJMj8msBJVPRmv7l6puNaKl42+1I2q2F12Q67uZh3SB12WTEtVDPKNF5uyfWop0JR w/KoXmtLhl/deLhMmK2JAuAFyHCcEy48EsRLiJcAojFti8gDfv6fw+QoFJqaMoP+ieGD S9wpiidjAINrZZkjcByoheaByykK/436ynSRDy1h9DqtB0NDf3B8IEFem+NM0rnvsQqp 9bHAIYBLQc/ulux68RnXU7tywYf78fBZ0m3fUvuB1SmRid4TnN4uIwGkRgEYfiNIbWFm +5kB3of1fCYryvl1e0Z+SUBEsAE9Yf8U5bB98hHcRm8aubeLwX3wj/u1hSjRm+v/3rlL LkUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726787788; x=1727392588; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=KVQBjSrLfeVCjZ7sTnJ78+yxGiIQHaOkWqkjn6pjxFQ=; b=TS3soMadZq2P3HCkgEiDVWugICHBRe0vKYzhH9DZm2PhO+wVTthUrYu9VgUDDCxDMR mjXH/hyAEVyeWfmyge8ek6IPk411PticC4twlwgY/K+IqbPwY62SBbkKKH6QLvT77Lpw UVr7Zn1xbKPLXjqFzhyMyY3nWTS0qKnVCq6Y0SeJWXYeFycOvbEbgcEA24JICbcep3bW TZ+3Por4CpNLD0LV/IJUda6i5cTawOD3ywtwqkHhjpYbwM/2geG1pikRORfd6FwuwnNT jS3yjQSw6hsAMNSn4VklK9q1MfukF0HUf/pgy16lpuRqOdKSnl+Wix9WZTKbmxtFThVi v4Rw== X-Gm-Message-State: AOJu0Yx2zGFd7c/DarJyjSb/s1u0WDMkTXA2jQTuDfPD4+Vk90dzm8JD 6qVP4Da2dMAKE/1JbUrGMmj0xPa+Ph3vbEDRhvpSIJuAbfShgYNw1quq10R17pAj6ZjrZSm2+gs 7Ps+CNMJBvUXNHUT82gRg5RQ97ya7aal1 X-Received: by 2002:a05:6e02:1e07:b0:39f:4e36:4b93 with SMTP id e9e14a558f8ab-3a0bef53e44mr36394025ab.6.1726787787829; Thu, 19 Sep 2024 16:16:27 -0700 (PDT) MIME-Version: 1.0 From: Carlos Ruiz Date: Fri, 20 Sep 2024 01:16:16 +0200 Message-ID: To: ffmpeg-devel@ffmpeg.org X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [FFmpeg-devel] [PATCH] avcodec/nvdec: support resizing while decoding X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: KuwMu3LQzuKp Hi! This is my first contribution to the project so please excuse any bad etiquette, I tried to read all the FAQs before posting. Would love to start by thanking everyone for such an amazing framework you've built! Anyway, here's my proposed patch to support video resizing when using NVDEC hwaccel to decode hevc video (I could look into a similar patch for h264, av1, etc if this looks useful). There's a bit more context/explanation in the commit description in the patch, but please let me know if the use case isn't clear. I tested locally and all of these 4 scenarios work as expected: * Using hevc codec with nvdec hwaccel, leaving avctx->width and avctx->height unset. On a 1920x1080 input video, I get 1920x1080 cuda frames out. * Using hevc codec with nvdec hwaccel, setting avctx->width and avctx->height to some arbitrary value (e.g. 640x360). On the same input video, I get 640x360 cuda frames out. * Using hevc codec without hwaccel, leaving avctx->width and avctx->height unset. I get 1920x1080 yuvj420p frames (in cpu) out. * Using hevc codec without hwaccel, setting avctx->width and avctx->height to some arbitrary value (e.g. 640x360). The values get ignored (as in FFMpeg master) and I again get 1920x1080 yuvj420p frames out. I'm not extremely familiar with hevcdec.c so I'm not sure if this would accidentally break something else. Looking forward to hearing your thoughts! From 850afda5f6479064c75a4b905f12e48f97b6d551 Mon Sep 17 00:00:00 2001 From: Carlos Ruiz Date: Thu, 19 Sep 2024 14:00:05 +0200 Subject: [PATCH] avcodec/nvdec: support resizing while decoding Nvidia chips support accelerated resizing while decoding video. The *_cuvid codecs (cuviddec.c) already support resizing and cropping, but have two big downsides: 1) they have a minimum latency of two packets (even with the LOW_DELAY flag enabled) 2) AV_CODEC_FLAG_COPY_OPAQUE is not respected (opaque and opaque_ref aren't transferred from packets to frames) Instead, parsing the video using a non-accelerated codec (hevcdec.c) solves both downsides above. This commit brings resizing capabilities to the *_nvdec hwaccel, similar to what *_cuvid does, to combine the best of both worlds (proper parsing + accelerated decoding and resizing). --- libavcodec/hevc/hevcdec.c | 8 ++++++-- libavcodec/nvdec.c | 21 +++++++++++++++++---- 2 files changed, 23 insertions(+), 6 deletions(-) + int ff_nvdec_decode_init(AVCodecContext *avctx) { NVDECContext *ctx = avctx->internal->hwaccel_priv_data; @@ -393,8 +405,9 @@ int ff_nvdec_decode_init(AVCodecContext *avctx) params.ulWidth = avctx->coded_width; params.ulHeight = avctx->coded_height; - params.ulTargetWidth = avctx->coded_width; - params.ulTargetHeight = avctx->coded_height; + avctx->get_buffer2 = get_buffer2; + params.ulTargetWidth = avctx->width; + params.ulTargetHeight = avctx->height; params.bitDepthMinus8 = sw_desc->comp[0].depth - 8; params.OutputFormat = output_format; params.CodecType = cuvid_codec_type; @@ -719,8 +732,8 @@ int ff_nvdec_frame_params(AVCodecContext *avctx, chroma_444 = supports_444 && cuvid_chroma_format == cudaVideoChromaFormat_444; frames_ctx->format = AV_PIX_FMT_CUDA; - frames_ctx->width = (avctx->coded_width + 1) & ~1; - frames_ctx->height = (avctx->coded_height + 1) & ~1; + frames_ctx->width = (avctx->width + 1) & ~1; + frames_ctx->height = (avctx->height + 1) & ~1; /* * We add two extra frames to the pool to account for deinterlacing filters * holding onto their frames. -- 2.43.0 diff --git a/libavcodec/hevc/hevcdec.c b/libavcodec/hevc/hevcdec.c index d915d74d22..d63fc5875f 100644 --- a/libavcodec/hevc/hevcdec.c +++ b/libavcodec/hevc/hevcdec.c @@ -351,8 +351,12 @@ static void export_stream_params(HEVCContext *s, const HEVCSPS *sps) avctx->pix_fmt = sps->pix_fmt; avctx->coded_width = sps->width; avctx->coded_height = sps->height; - avctx->width = sps->width - ow->left_offset - ow->right_offset; - avctx->height = sps->height - ow->top_offset - ow->bottom_offset; + if (avctx->width <= 0 || avctx->height <= 0) { + avctx->width = sps->width; + avctx->height = sps->height; + } + avctx->width = avctx->width - ow->left_offset - ow->right_offset; + avctx->height = avctx->height - ow->top_offset - ow->bottom_offset; avctx->has_b_frames = sps->temporal_layer[sps->max_sub_layers - 1].num_reorder_pics; avctx->profile = sps->ptl.general_ptl.profile_idc; avctx->level = sps->ptl.general_ptl.level_idc; diff --git a/libavcodec/nvdec.c b/libavcodec/nvdec.c index 932544564a..86143de74c 100644 --- a/libavcodec/nvdec.c +++ b/libavcodec/nvdec.c @@ -324,6 +324,18 @@ static int nvdec_init_hwframes(AVCodecContext *avctx, AVBufferRef **out_frames_r return 0; } +static int get_buffer2(AVCodecContext *avctx, AVFrame *frame, int flags) { + /* + * HEVC codec includes FF_CODEC_CAP_EXPORTS_CROPPING in its caps_internal, so by default frames will be set + * to width=avctx->coded_width and height=avctx->coded_height. Now that we support resizing as part of decoding, + * overwrite the frame dimensions with display values rather than coded. + */ + int ret = avcodec_default_get_buffer2(avctx, frame, flags); + frame->width = avctx->width; + frame->height = avctx->height; + return ret; +}