From patchwork Mon Sep 9 10:37:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 51473 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:9c4f:0:b0:48e:c0f8:d0de with SMTP id w15csp1543975vqu; Mon, 9 Sep 2024 03:38:34 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUlCZ1zF/2exG9ttpLxWG3MmSIKott7rqp2iA1FR9OqEQLHfUrxIyu+HxVbPPiq/G8zDnDOuE8Snt9/RA34cZj2@gmail.com X-Google-Smtp-Source: AGHT+IF7smu+TA36uNWWhwqcvYQj4QXw29pcyU1JXX0Jc42YDPkKHuwbcwZho7fgx1AL5Wq02Hl4 X-Received: by 2002:a05:6512:3c96:b0:52f:c2b6:dcfe with SMTP id 2adb3069b0e04-5365856a178mr3563472e87.0.1725878314402; Mon, 09 Sep 2024 03:38:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1725878314; cv=none; d=google.com; s=arc-20240605; b=IOgkaveDFvNnMEnv3FOB6cFhv6n7w4EbEQF3gM5u0n24lqyjg/Bh+Uqy6pVNvaWNU/ wqVLsspQOnAMrDGW+CbibpPDzVz13ipVCjH6+POHhhZJj3nKhB962TRVqzzdU4J5C33J oiwIR49NgwRTnzq40pmEEOwlabDw9NwHmFqvmQoNb4B4Lskit5772QZ2wWnCWitx5XpW o4LcTaiyGx9g1G+cAggJ+gXOwM612Bm+YuCYZ+s6olhQLAfCcV18YZj43eHk0HIA6uNm L8oh28ofawNnWoHcDNOe5du3PprYN5r1STxsM+U29yTEjvnUaqTIlB+rIbCqvbC27Adm legQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to :delivered-to; bh=C2rxruqxcm3z2SEXOm8KSMQSyzYUi7GQpqCS5WDtSto=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=KSe+hg7hdgfw3OvrUnCHBVlQUedi+02nD98vhdOztMng29541Sh3bWSRe41YmPVvCa j5jBSbTtUZIEGfmIf61q21UX9WauEcecOw95Eg2j1rDdEqgsxRlzfl1Sf/tXXWE+P68y qcsecdVMV8vZWakJldoXDt5SC5db3hB8SIl+66Du1cLF2Pi9mXXstikL4Kejr+H0jY7G WUaafpr3K6wqrOANBQeagom6r5IPLvsCGr6R9wAmdwejvz8Don0PHcXZqVl8ZZDCGHOZ 21lYNpUgutCfZrl20szMDvPTpfp1ZCW5NuCHkXkO0VUYzYyTdMg6QEkB2hYZK11aGK+z 2vxw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-5365f866ebasi1578232e87.97.2024.09.09.03.38.33; Mon, 09 Sep 2024 03:38:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 88E6168DF4F; Mon, 9 Sep 2024 13:38:29 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id CBC5068D96F for ; Mon, 9 Sep 2024 13:38:22 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Mon, 9 Sep 2024 12:37:51 +0200 Message-ID: <20240909103759.371919-1-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] lavc: add Vulkan video encoding base code X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: t9L9Ng+2WXLJ This commit adds the common Vulkan video encoding framework. It makes full use of the asynchronous features of our new common hardware encoding code, and of Vulkan. The code is able to handle anything from H264 to AV1 and MJPEG. --- configure | 2 + libavcodec/Makefile | 2 +- libavcodec/vulkan_encode.c | 979 +++++++++++++++++++++++++++++++++++++ libavcodec/vulkan_encode.h | 243 +++++++++ 4 files changed, 1225 insertions(+), 1 deletion(-) create mode 100644 libavcodec/vulkan_encode.c create mode 100644 libavcodec/vulkan_encode.h diff --git a/configure b/configure index a8e67d230c..6cfb736a86 100755 --- a/configure +++ b/configure @@ -2638,6 +2638,7 @@ CONFIG_EXTRA=" vp3dsp vp56dsp vp8dsp + vulkan_encode wma_freqs wmv2dsp " @@ -3299,6 +3300,7 @@ qsvdec_select="qsv" qsvenc_select="qsv" qsvvpp_select="qsv" vaapi_encode_deps="vaapi" +vulkan_encode_deps="vulkan" v4l2_m2m_deps="linux_videodev2_h sem_timedwait" bilateral_cuda_filter_deps="ffnvcodec" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 27ef4638ce..ff6a3c4efc 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -1282,7 +1282,7 @@ SKIPHEADERS-$(CONFIG_QSVENC) += qsvenc.h SKIPHEADERS-$(CONFIG_VAAPI) += vaapi_decode.h vaapi_hevc.h vaapi_encode.h SKIPHEADERS-$(CONFIG_VDPAU) += vdpau.h vdpau_internal.h SKIPHEADERS-$(CONFIG_VIDEOTOOLBOX) += videotoolbox.h vt_internal.h -SKIPHEADERS-$(CONFIG_VULKAN) += vulkan.h vulkan_video.h vulkan_decode.h +SKIPHEADERS-$(CONFIG_VULKAN) += vulkan.h vulkan_video.h vulkan_encode.h vulkan_decode.h SKIPHEADERS-$(CONFIG_V4L2_M2M) += v4l2_buffers.h v4l2_context.h v4l2_m2m.h SKIPHEADERS-$(CONFIG_ZLIB) += zlib_wrapper.h diff --git a/libavcodec/vulkan_encode.c b/libavcodec/vulkan_encode.c new file mode 100644 index 0000000000..5e87d4c073 --- /dev/null +++ b/libavcodec/vulkan_encode.c @@ -0,0 +1,979 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/mem.h" +#include "libavutil/avassert.h" +#include "vulkan_encode.h" +#include "config.h" + +#include "libavutil/vulkan_loader.h" + +const AVCodecHWConfigInternal *const ff_vulkan_encode_hw_configs[] = { + HW_CONFIG_ENCODER_FRAMES(VULKAN, VULKAN), + NULL, +}; + +av_cold void ff_vulkan_encode_uninit(FFVulkanEncodeContext *ctx) +{ + FFVulkanContext *s = &ctx->s; + FFVulkanFunctions *vk = &s->vkfn; + + /* Wait on and free execution pool */ + ff_vk_exec_pool_free(s, &ctx->enc_pool); + + /* Destroy the session params */ + if (ctx->session_params) + vk->DestroyVideoSessionParametersKHR(s->hwctx->act_dev, + ctx->session_params, + s->hwctx->alloc); + + ff_hw_base_encode_close(&ctx->base); + + av_buffer_pool_uninit(&ctx->buf_pool); + + ff_vk_video_common_uninit(s, &ctx->common); + + ff_vk_uninit(s); +} + +static int vulkan_encode_init(AVCodecContext *avctx, FFHWBaseEncodePicture *pic) +{ + int err; + FFVulkanEncodeContext *ctx = avctx->priv_data; + FFVulkanEncodePicture *vp = pic->priv; + + AVFrame *f = pic->input_image; + AVHWFramesContext *hwfc = (AVHWFramesContext *)f->hw_frames_ctx->data; + AVVulkanFramesContext *vkfc = hwfc->hwctx; + AVVkFrame *vkf = (AVVkFrame *)f->data[0]; + + if (ctx->codec->picture_priv_data_size > 0) { + pic->codec_priv = av_mallocz(ctx->codec->picture_priv_data_size); + if (!pic->codec_priv) + return AVERROR(ENOMEM); + } + + /* Input image view */ + err = ff_vk_create_view(&ctx->s, &ctx->common, + &vp->in.view, &vp->in.aspect, + vkf, vkfc->format[0], 0); + if (err < 0) + return err; + + /* Reference view */ + if (!ctx->common.layered_dpb) { + AVFrame *rf = pic->recon_image; + AVVkFrame *rvkf = (AVVkFrame *)rf->data[0]; + err = ff_vk_create_view(&ctx->s, &ctx->common, + &vp->dpb.view, &vp->dpb.aspect, + rvkf, ctx->pic_format, 1); + if (err < 0) + return err; + } else { + vp->dpb.view = ctx->common.layered_view; + vp->dpb.aspect = ctx->common.layered_aspect; + } + + return 0; +} + +static int vulkan_encode_free(AVCodecContext *avctx, FFHWBaseEncodePicture *pic) +{ + FFVulkanEncodeContext *ctx = avctx->priv_data; + FFVulkanFunctions *vk = &ctx->s.vkfn; + + FFVulkanEncodePicture *vp = pic->priv; + + if (vp->in.view) + vk->DestroyImageView(ctx->s.hwctx->act_dev, vp->in.view, + ctx->s.hwctx->alloc); + + if (!ctx->common.layered_dpb && vp->dpb.view) + vk->DestroyImageView(ctx->s.hwctx->act_dev, vp->dpb.view, + ctx->s.hwctx->alloc); + + ctx->slots[vp->dpb_slot.slotIndex] = 0; + + return 0; +} + +static int init_pic_rc(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + VkVideoEncodeRateControlInfoKHR *rc_info, + VkVideoEncodeRateControlLayerInfoKHR *rc_layer /* Goes in ^ */) +{ + FFVulkanEncodeContext *ctx = avctx->priv_data; + + *rc_info = (VkVideoEncodeRateControlInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_RATE_CONTROL_INFO_KHR, + .rateControlMode = ctx->opts.rc_mode, + }; + + if (ctx->opts.rc_mode > VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR) { + *rc_layer = (VkVideoEncodeRateControlLayerInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_RATE_CONTROL_LAYER_INFO_KHR, + .averageBitrate = avctx->bit_rate, + .maxBitrate = avctx->rc_max_rate ? avctx->rc_max_rate : avctx->bit_rate, + .frameRateNumerator = avctx->framerate.num, + .frameRateDenominator = avctx->framerate.den, + }; + rc_info->layerCount++; + rc_info->pLayers = rc_layer; + } + + return ctx->codec->init_pic_rc(avctx, pic, rc_info, rc_layer); +} + +static int vulkan_encode_issue(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic) +{ + FFVulkanEncodeContext *ctx = avctx->priv_data; + FFVulkanFunctions *vk = &ctx->s.vkfn; + + const size_t size_align = ctx->caps.minBitstreamBufferSizeAlignment; + + FFVulkanEncodePicture *vp = base_pic->priv; + AVFrame *src = (AVFrame *)base_pic->input_image; + AVVkFrame *vkf = (AVVkFrame *)src->data[0]; + + int err, max_pkt_size; + + FFVkBuffer *sd_buf; + + int slot_index = -1; + FFVkExecContext *exec; + VkCommandBuffer cmd_buf; + VkImageMemoryBarrier2 img_bar[37]; + int nb_img_bar = 0; + + /* Coding start/end */ + VkVideoBeginCodingInfoKHR encode_start; + VkVideoEndCodingInfoKHR encode_end = { + .sType = VK_STRUCTURE_TYPE_VIDEO_END_CODING_INFO_KHR, + }; + + VkVideoEncodeRateControlLayerInfoKHR rc_layer; + VkVideoEncodeRateControlInfoKHR rc_info; + VkVideoEncodeQualityLevelInfoKHR q_info; + VkVideoCodingControlInfoKHR encode_ctrl; + + VkVideoReferenceSlotInfoKHR ref_slot[37]; + VkVideoEncodeInfoKHR encode_info; + + /* Create packet data buffer */ + max_pkt_size = FFALIGN(3 * ctx->base.surface_width * ctx->base.surface_height + (1 << 16), + ctx->caps.minBitstreamBufferSizeAlignment); + + err = ff_vk_get_pooled_buffer(&ctx->s, &ctx->buf_pool, &vp->pkt_buf, + VK_BUFFER_USAGE_VIDEO_ENCODE_DST_BIT_KHR, + &ctx->profile_list, max_pkt_size, + VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | + VK_MEMORY_PROPERTY_HOST_CACHED_BIT); + if (err < 0) + return err; + + sd_buf = (FFVkBuffer *)vp->pkt_buf->data; + + /* Setup rate control */ + err = init_pic_rc(avctx, base_pic, &rc_info, &rc_layer); + if (err < 0) + return err; + + q_info = (VkVideoEncodeQualityLevelInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_QUALITY_LEVEL_INFO_KHR, + .pNext = &rc_info, + .qualityLevel = ctx->opts.quality, + }; + encode_ctrl = (VkVideoCodingControlInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_CODING_CONTROL_INFO_KHR, + .pNext = &q_info, + .flags = VK_VIDEO_CODING_CONTROL_ENCODE_QUALITY_LEVEL_BIT_KHR | + VK_VIDEO_CODING_CONTROL_ENCODE_RATE_CONTROL_BIT_KHR | + (base_pic->force_idr ? VK_VIDEO_CODING_CONTROL_RESET_BIT_KHR : 0), + }; + + /* Current picture's ref slot */ + vp->dpb_res = (VkVideoPictureResourceInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_PICTURE_RESOURCE_INFO_KHR, + .pNext = NULL, + .codedOffset = { 0 }, + .codedExtent = (VkExtent2D){ ctx->base.surface_width, + ctx->base.surface_height }, + .baseArrayLayer = 0, + .imageViewBinding = vp->dpb.view, + }; + + for (int i = 0; i < ctx->caps.maxDpbSlots; i++) { + if (!ctx->slots[i]) { + slot_index = i; + ctx->slots[i] = 1; + break; + } + } + av_assert0(slot_index >= 0); + + vp->dpb_slot = (VkVideoReferenceSlotInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_REFERENCE_SLOT_INFO_KHR, + .pNext = NULL, // Set later + .slotIndex = slot_index, + .pPictureResource = &vp->dpb_res, + }; + + encode_info = (VkVideoEncodeInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_INFO_KHR, + .pNext = NULL, // Set later + .flags = 0x0, + .srcPictureResource = (VkVideoPictureResourceInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_PICTURE_RESOURCE_INFO_KHR, + .pNext = NULL, + .codedOffset = { 0, 0 }, + .codedExtent = (VkExtent2D){ base_pic->input_image->width, + base_pic->input_image->height }, + .baseArrayLayer = 0, + .imageViewBinding = vp->in.view, + }, + .pSetupReferenceSlot = &vp->dpb_slot, + .referenceSlotCount = 0, + .pReferenceSlots = ref_slot, + .dstBuffer = sd_buf->buf, + .dstBufferOffset = 0, + .dstBufferRange = sd_buf->size, + .precedingExternallyEncodedBytes = 0, + }; + + for (int i = 0; i < MAX_REFERENCE_LIST_NUM; i++) { + for (int j = 0; j < base_pic->nb_refs[i]; j++) { + FFHWBaseEncodePicture *ref = base_pic->refs[i][j]; + FFVulkanEncodePicture *rvp = ref->priv; + ref_slot[encode_info.referenceSlotCount++] = rvp->dpb_slot; + } + } + + /* Setup picture parameters */ + err = ctx->codec->init_pic_params(avctx, base_pic, + &encode_info); + if (err < 0) + return err; + + encode_start = (VkVideoBeginCodingInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_BEGIN_CODING_INFO_KHR, + .pNext = !base_pic->force_idr ? &rc_info : NULL, + .videoSession = ctx->common.session, + .videoSessionParameters = ctx->session_params, + .referenceSlotCount = encode_info.referenceSlotCount, + .pReferenceSlots = ref_slot, + }; + + /* Calling vkCmdBeginVideoCodingKHR requires to declare all references + * being enabled upfront, including the current frame's output ref. + * If layered DBPs are used, make sure its not included twice. */ + if (!ctx->common.layered_dpb || !encode_info.referenceSlotCount) { + ref_slot[encode_info.referenceSlotCount] = vp->dpb_slot; + ref_slot[encode_info.referenceSlotCount].slotIndex = -1; + encode_start.referenceSlotCount++; + } + + /* Write header */ + if (base_pic->type == FF_HW_PICTURE_TYPE_IDR) { + uint8_t *hdr_dst = sd_buf->mapped_mem + encode_info.dstBufferOffset; + size_t data_size = encode_info.dstBufferRange; + err = ctx->codec->write_sequence_headers(avctx, base_pic, hdr_dst, &data_size); + if (err < 0) + goto fail; + encode_info.dstBufferOffset += data_size; + encode_info.dstBufferRange -= data_size; + } + + /* Write extra units */ + if (ctx->codec->write_extra_headers) { + uint8_t *hdr_dst = sd_buf->mapped_mem + encode_info.dstBufferOffset; + size_t data_size = encode_info.dstBufferRange; + err = ctx->codec->write_extra_headers(avctx, base_pic, hdr_dst, &data_size); + if (err < 0) + goto fail; + encode_info.dstBufferOffset += data_size; + encode_info.dstBufferRange -= data_size; + } + + /* Align buffer offset to the required value with filler units */ + if (ctx->codec->write_filler) { + uint8_t *hdr_dst = sd_buf->mapped_mem + encode_info.dstBufferOffset; + size_t data_size = encode_info.dstBufferRange; + + uint32_t offset = encode_info.dstBufferOffset; + size_t offset_align = ctx->caps.minBitstreamBufferOffsetAlignment; + + uint32_t filler_data = FFALIGN(offset, offset_align) - offset; + + if (filler_data) { + while (filler_data < ctx->codec->filler_header_size) + filler_data += offset_align; + + filler_data -= ctx->codec->filler_header_size; + + err = ctx->codec->write_filler(avctx, filler_data, + hdr_dst, &data_size); + if (err < 0) + goto fail; + + encode_info.dstBufferOffset += data_size; + encode_info.dstBufferRange -= data_size; + } + } + + vp->slices_offset = encode_info.dstBufferOffset; + + /* Align buffer size to the nearest lower alignment requirement. */ + encode_info.dstBufferRange -= size_align; + encode_info.dstBufferRange = FFALIGN(encode_info.dstBufferRange, + size_align); + + /* Start command buffer recording */ + exec = vp->exec = ff_vk_exec_get(&ctx->enc_pool); + ff_vk_exec_start(&ctx->s, exec); + cmd_buf = exec->buf; + + /* Output packet buffer */ + err = ff_vk_exec_add_dep_buf(&ctx->s, exec, &vp->pkt_buf, 1, 1); + if (err < 0) + goto fail; + + /* Source image */ + err = ff_vk_exec_add_dep_frame(&ctx->s, exec, src, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR); + if (err < 0) + goto fail; + + /* Source image layout conversion */ + img_bar[nb_img_bar] = (VkImageMemoryBarrier2) { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2, + .pNext = NULL, + .srcStageMask = VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + .srcAccessMask = vkf->access[0], + .dstStageMask = VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR, + .dstAccessMask = VK_ACCESS_2_VIDEO_ENCODE_READ_BIT_KHR, + .oldLayout = vkf->layout[0], + .newLayout = VK_IMAGE_LAYOUT_VIDEO_ENCODE_SRC_KHR, + .srcQueueFamilyIndex = vkf->queue_family[0], + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = vkf->img[0], + .subresourceRange = (VkImageSubresourceRange) { + .aspectMask = vp->in.aspect, + .layerCount = 1, + .levelCount = 1, + }, + }; + ff_vk_exec_update_frame(&ctx->s, exec, src, + &img_bar[nb_img_bar], &nb_img_bar); + + if (!ctx->common.layered_dpb) { + /* Source image's ref slot. + * No need to do a layout conversion, since the frames which are allocated + * with a DPB usage are automatically converted. */ + err = ff_vk_exec_add_dep_frame(&ctx->s, exec, base_pic->recon_image, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR); + if (err < 0) + return err; + + /* All references */ + for (int i = 0; i < MAX_REFERENCE_LIST_NUM; i++) { + for (int j = 0; j < base_pic->nb_refs[i]; j++) { + FFHWBaseEncodePicture *ref = base_pic->refs[i][j]; + err = ff_vk_exec_add_dep_frame(&ctx->s, exec, ref->recon_image, + VK_PIPELINE_STAGE_2_ALL_COMMANDS_BIT, + VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR); + if (err < 0) + return err; + } + } + } else { + err = ff_vk_exec_add_dep_frame(&ctx->s, exec, ctx->common.layered_frame, + VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR, + VK_PIPELINE_STAGE_2_VIDEO_ENCODE_BIT_KHR); + if (err < 0) + return err; + } + + /* Change image layout */ + vk->CmdPipelineBarrier2(cmd_buf, &(VkDependencyInfo) { + .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO, + .pImageMemoryBarriers = img_bar, + .imageMemoryBarrierCount = nb_img_bar, + }); + + /* Start, use parameters */ + vk->CmdBeginVideoCodingKHR(cmd_buf, &encode_start); + + /* Send control data */ + if (base_pic->force_idr) + vk->CmdControlVideoCodingKHR(cmd_buf, &encode_ctrl); + + /* Encode */ + vk->CmdBeginQuery(cmd_buf, ctx->enc_pool.query_pool, exec->query_idx + 0, 0); + vk->CmdEncodeVideoKHR(cmd_buf, &encode_info); + vk->CmdEndQuery(cmd_buf, ctx->enc_pool.query_pool, exec->query_idx + 0); + + /* End encoding */ + vk->CmdEndVideoCodingKHR(cmd_buf, &encode_end); + + /* End recording and submit for execution */ + err = ff_vk_exec_submit(&ctx->s, vp->exec); + if (err < 0) + goto fail; + + /* We don't need to keep the input image any longer, its already ref'd */ + av_frame_free(&base_pic->input_image); + + return 0; + +fail: + return err; +} + +static void vulkan_encode_wait(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic) +{ + FFVulkanEncodeContext *ctx = avctx->priv_data; + FFVulkanEncodePicture *vp = base_pic->priv; + + av_assert0(base_pic->encode_issued); + + if (base_pic->encode_complete) + return; + + ff_vk_exec_wait(&ctx->s, vp->exec); + base_pic->encode_complete = 1; +} + +static int vulkan_encode_output(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic, AVPacket *pkt) +{ + VkResult ret; + FFVulkanEncodePicture *vp = base_pic->priv; + FFVulkanEncodeContext *ctx = avctx->priv_data; + FFVkBuffer *sd_buf = (FFVkBuffer *)vp->pkt_buf->data; + uint32_t *query_data; + + vulkan_encode_wait(avctx, base_pic); + + ret = ff_vk_exec_get_query(&ctx->s, vp->exec, (void **)&query_data, 0); + if (ret == VK_NOT_READY) { + av_log(avctx, AV_LOG_ERROR, "Unable to perform query: %s!\n", + ff_vk_ret2str(ret)); + return AVERROR(EINVAL); + } + + if (ret != VK_NOT_READY && ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Unable to perform query: %s!\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + if (query_data[2] != VK_QUERY_RESULT_STATUS_COMPLETE_KHR) { + av_log(avctx, AV_LOG_ERROR, "Unable to encode: %u\n", query_data[2]); + return AVERROR_EXTERNAL; + } + + /* Invalidate buffer if needed */ + if (!(sd_buf->flags & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) { + FFVulkanFunctions *vk = &ctx->s.vkfn; + VkMappedMemoryRange invalidate_buf; + + int offs = vp->slices_offset; + /* If the non-coherent alignment is greater than the bitstream buffer + * offset's alignment, and the offs value is not aligned already, + * align it to the previous alignment point. */ + if (ctx->s.props.properties.limits.nonCoherentAtomSize > + ctx->caps.minBitstreamBufferOffsetAlignment && offs && + (FFALIGN(offs, ctx->s.props.properties.limits.nonCoherentAtomSize) != offs)) { + offs -= ctx->s.props.properties.limits.nonCoherentAtomSize; + offs = FFALIGN(FFMAX(offs, 0), ctx->s.props.properties.limits.nonCoherentAtomSize); + } + + invalidate_buf = (VkMappedMemoryRange) { + .sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE, + .memory = sd_buf->mem, + .offset = offs, + .size = VK_WHOLE_SIZE, + }; + + vk->FlushMappedMemoryRanges(ctx->s.hwctx->act_dev, 1, &invalidate_buf); + } + + pkt->data = sd_buf->mapped_mem; + pkt->size = vp->slices_offset + /* base offset */ + query_data[0] /* secondary offset */ + + query_data[1] /* size */; + + /* Move reference */ + pkt->buf = vp->pkt_buf; + vp->pkt_buf = NULL; + + av_log(avctx, AV_LOG_DEBUG, "Frame %"PRId64"/%"PRId64 " encoded\n", + base_pic->display_order, base_pic->encode_order); + + ff_hw_base_encode_set_output_property(&ctx->base, avctx, + base_pic, pkt, + ctx->codec->flags & VK_ENC_FLAG_NO_DELAY); + + return 0; +} + +static const FFHWEncodePictureOperation vulkan_base_encode_ops = { + .priv_size = sizeof(FFVulkanEncodePicture), + .init = &vulkan_encode_init, + .issue = &vulkan_encode_issue, + .output = &vulkan_encode_output, + .free = &vulkan_encode_free, +}; + +int ff_vulkan_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt) +{ + FFVulkanEncodeContext *ctx = avctx->priv_data; + return ff_hw_base_encode_receive_packet(&ctx->base, avctx, pkt); +} + +static int vulkan_encode_create_dpb(AVCodecContext *avctx, FFVulkanEncodeContext *ctx) +{ + int err; + FFHWBaseEncodeContext *base_ctx = &ctx->base; + AVVulkanFramesContext *hwfc; + + enum AVPixelFormat dpb_format; + err = ff_hw_base_get_recon_format(base_ctx, NULL, &dpb_format); + if (err < 0) + return err; + + base_ctx->recon_frames_ref = av_hwframe_ctx_alloc(base_ctx->device_ref); + if (!base_ctx->recon_frames_ref) + return AVERROR(ENOMEM); + + base_ctx->recon_frames = (AVHWFramesContext *)base_ctx->recon_frames_ref->data; + hwfc = (AVVulkanFramesContext *)base_ctx->recon_frames->hwctx; + + base_ctx->recon_frames->format = AV_PIX_FMT_VULKAN; + base_ctx->recon_frames->sw_format = dpb_format; + base_ctx->recon_frames->width = base_ctx->surface_width; + base_ctx->recon_frames->height = base_ctx->surface_height; + + hwfc->format[0] = ctx->pic_format; + hwfc->create_pnext = &ctx->profile_list; + hwfc->tiling = VK_IMAGE_TILING_OPTIMAL; + hwfc->usage = VK_IMAGE_USAGE_SAMPLED_BIT | + VK_IMAGE_USAGE_VIDEO_ENCODE_DPB_BIT_KHR; + + if (ctx->common.layered_dpb) + hwfc->nb_layers = ctx->caps.maxDpbSlots; + + err = av_hwframe_ctx_init(base_ctx->recon_frames_ref); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to initialise DPB frame context: %s\n", + av_err2str(err)); + return err; + } + + if (ctx->common.layered_dpb) { + ctx->common.layered_frame = av_frame_alloc(); + if (!ctx->common.layered_frame) + return AVERROR(ENOMEM); + + err = av_hwframe_get_buffer(base_ctx->recon_frames_ref, + ctx->common.layered_frame, 0); + if (err < 0) + return AVERROR(ENOMEM); + + err = ff_vk_create_view(&ctx->s, &ctx->common, + &ctx->common.layered_view, + &ctx->common.layered_aspect, + (AVVkFrame *)ctx->common.layered_frame->data[0], + hwfc->format[0], 1); + if (err < 0) + return err; + + av_buffer_unref(&base_ctx->recon_frames_ref); + } + + return 0; +} + +static av_cold int init_rc(AVCodecContext *avctx, FFVulkanEncodeContext *ctx) +{ + if (ctx->opts.rc_mode == FF_VK_RC_MODE_AUTO) { + if (ctx->opts.qp >= 0) { + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR; + } else if (avctx->global_quality > 0) { + if (avctx->flags & AV_CODEC_FLAG_QSCALE) + ctx->opts.qp = avctx->global_quality / FF_QP2LAMBDA; + else + ctx->opts.qp = avctx->global_quality; + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR; + } else if (avctx->bit_rate) { + if (ctx->enc_caps.rateControlModes & VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR) + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR; + else if (ctx->enc_caps.rateControlModes & VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR) + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR; + else + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHR; + } else { + ctx->opts.qp = 18; + ctx->opts.rc_mode = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR; + av_log(avctx, AV_LOG_WARNING, "No rate control settings specified, using fixed QP = %i\n", + ctx->opts.qp); + } + } + + if (ctx->opts.rc_mode && !(ctx->enc_caps.rateControlModes & ctx->opts.rc_mode)) { + static const char *rc_modes[] = { + [VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHR] = "default", + [VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR] = "cqp", + [VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR] = "cbr", + [VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR] = "vbr", + }; + av_log(avctx, AV_LOG_ERROR, "Unsupported rate control mode %i, supported are:\n", + ctx->opts.rc_mode); + for (int i = 0; i < av_popcount(ctx->enc_caps.rateControlModes); i++) { + if (!(ctx->enc_caps.rateControlModes & (1 << i))) + continue; + av_log(avctx, AV_LOG_ERROR, " %i: %s\n", i, rc_modes[i]); + } + return AVERROR(ENOTSUP); + } + + return 0; +} + +av_cold int ff_vulkan_encode_init(AVCodecContext *avctx, FFVulkanEncodeContext *ctx, + const FFVulkanEncodeDescriptor *vk_desc, + const FFVulkanCodec *codec, + void *codec_caps, void *quality_pnext) +{ + int i, err; + VkResult ret; + FFVulkanFunctions *vk = &ctx->s.vkfn; + FFVulkanContext *s = &ctx->s; + FFHWBaseEncodeContext *base_ctx = &ctx->base; + + const AVPixFmtDescriptor *desc; + + VkVideoFormatPropertiesKHR *ret_info; + uint32_t nb_out_fmts = 0; + + VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR quality_info; + + VkQueryPoolVideoEncodeFeedbackCreateInfoKHR query_create; + + VkVideoSessionCreateInfoKHR session_create = { + .sType = VK_STRUCTURE_TYPE_VIDEO_SESSION_CREATE_INFO_KHR, + }; + VkPhysicalDeviceVideoFormatInfoKHR fmt_info = { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VIDEO_FORMAT_INFO_KHR, + .pNext = &ctx->profile_list, + }; + + if (!avctx->hw_frames_ctx) { + av_log(avctx, AV_LOG_ERROR, "A hardware frames reference is " + "required to associate the encoding device.\n"); + return AVERROR(EINVAL); + } + + ctx->base.op = &vulkan_base_encode_ops; + ctx->codec = codec; + + s->frames_ref = av_buffer_ref(avctx->hw_frames_ctx); + s->frames = (AVHWFramesContext *)s->frames_ref->data; + s->hwfc = s->frames->hwctx; + + s->device = (AVHWDeviceContext *)s->frames->device_ref->data; + s->hwctx = s->device->hwctx; + + desc = av_pix_fmt_desc_get(avctx->sw_pix_fmt); + if (!desc) + return AVERROR(EINVAL); + + s->extensions = ff_vk_extensions_to_mask(s->hwctx->enabled_dev_extensions, + s->hwctx->nb_enabled_dev_extensions); + + if (!(s->extensions & FF_VK_EXT_VIDEO_ENCODE_QUEUE)) { + av_log(avctx, AV_LOG_ERROR, "Device does not support the %s extension!\n", + VK_KHR_VIDEO_ENCODE_QUEUE_EXTENSION_NAME); + return AVERROR(ENOSYS); + } else if (!(s->extensions & FF_VK_EXT_VIDEO_MAINTENANCE_1)) { + av_log(avctx, AV_LOG_ERROR, "Device does not support the %s extension!\n", + VK_KHR_VIDEO_MAINTENANCE_1_EXTENSION_NAME); + return AVERROR(ENOSYS); + } else if (!(s->extensions & vk_desc->encode_extension)) { + av_log(avctx, AV_LOG_ERROR, "Device does not support decoding %s!\n", + avcodec_get_name(avctx->codec_id)); + return AVERROR(ENOSYS); + } + + /* Load functions */ + err = ff_vk_load_functions(s->device, vk, s->extensions, 1, 1); + if (err < 0) + return err; + + /* Create queue context */ + err = ff_vk_video_qf_init(s, &ctx->qf_enc, + VK_QUEUE_VIDEO_ENCODE_BIT_KHR, + vk_desc->encode_op); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Decoding of %s is not supported by this device\n", + avcodec_get_name(avctx->codec_id)); + return err; + } + + /* Load all properties */ + err = ff_vk_load_props(s); + if (err < 0) + return err; + + /* Set tuning */ + ctx->usage_info = (VkVideoEncodeUsageInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_USAGE_INFO_KHR, + .videoUsageHints = ctx->opts.usage, + .videoContentHints = ctx->opts.content, + .tuningMode = ctx->opts.tune, + }; + + /* Load up the profile now, needed for caps and to create a query pool */ + ctx->profile.sType = VK_STRUCTURE_TYPE_VIDEO_PROFILE_INFO_KHR; + ctx->profile.pNext = &ctx->usage_info; + ctx->profile.videoCodecOperation = vk_desc->encode_op; + ctx->profile.chromaSubsampling = ff_vk_subsampling_from_av_desc(desc); + ctx->profile.lumaBitDepth = ff_vk_depth_from_av_depth(desc->comp[0].depth); + ctx->profile.chromaBitDepth = ctx->profile.lumaBitDepth; + + /* Setup a profile */ + err = codec->init_profile(avctx, &ctx->profile, &ctx->usage_info); + if (err < 0) + return err; + + ctx->profile_list.sType = VK_STRUCTURE_TYPE_VIDEO_PROFILE_LIST_INFO_KHR; + ctx->profile_list.profileCount = 1; + ctx->profile_list.pProfiles = &ctx->profile; + + /* Get the capabilities of the encoder for the given profile */ + ctx->enc_caps.sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_CAPABILITIES_KHR; + ctx->enc_caps.pNext = codec_caps; + ctx->caps.sType = VK_STRUCTURE_TYPE_VIDEO_CAPABILITIES_KHR; + ctx->caps.pNext = &ctx->enc_caps; + + ret = vk->GetPhysicalDeviceVideoCapabilitiesKHR(s->hwctx->phys_dev, + &ctx->profile, + &ctx->caps); + if (ret == VK_ERROR_VIDEO_PROFILE_OPERATION_NOT_SUPPORTED_KHR) { + av_log(avctx, AV_LOG_ERROR, "Unable to initialize encoding: " + "%s profile \"%s\" not supported!\n", + avcodec_get_name(avctx->codec_id), + avcodec_profile_name(avctx->codec_id, avctx->profile)); + return AVERROR(EINVAL); + } else if (ret == VK_ERROR_VIDEO_PROFILE_FORMAT_NOT_SUPPORTED_KHR) { + av_log(avctx, AV_LOG_ERROR, "Unable to initialize encoding: " + "format (%s) not supported!\n", + av_get_pix_fmt_name(avctx->sw_pix_fmt)); + return AVERROR(EINVAL); + } else if (ret == VK_ERROR_FEATURE_NOT_PRESENT || + ret == VK_ERROR_FORMAT_NOT_SUPPORTED) { + return AVERROR(EINVAL); + } else if (ret != VK_SUCCESS) { + return AVERROR_EXTERNAL; + } + + err = init_rc(avctx, ctx); + if (err < 0) + return err; + + /* Create command and query pool */ + query_create = (VkQueryPoolVideoEncodeFeedbackCreateInfoKHR) { + .sType = VK_STRUCTURE_TYPE_QUERY_POOL_VIDEO_ENCODE_FEEDBACK_CREATE_INFO_KHR, + .pNext = &ctx->profile, + .encodeFeedbackFlags = ctx->enc_caps.supportedEncodeFeedbackFlags, + }; + err = ff_vk_exec_pool_init(s, &ctx->qf_enc, &ctx->enc_pool, base_ctx->async_depth, + 1, VK_QUERY_TYPE_VIDEO_ENCODE_FEEDBACK_KHR, 0, + &query_create); + if (err < 0) + return err; + + if (ctx->opts.quality > ctx->enc_caps.maxQualityLevels) { + av_log(avctx, AV_LOG_ERROR, "Invalid quality level %i: allowed range is " + "0 to %i\n", + ctx->opts.quality, ctx->enc_caps.maxQualityLevels); + return AVERROR(EINVAL); + } + + /* Get quality properties for the profile and quality level */ + quality_info = (VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR) { + .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VIDEO_ENCODE_QUALITY_LEVEL_INFO_KHR, + .pVideoProfile = &ctx->profile, + .qualityLevel = ctx->opts.quality, + }; + ctx->quality_props = (VkVideoEncodeQualityLevelPropertiesKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_QUALITY_LEVEL_PROPERTIES_KHR, + .pNext = quality_pnext, + }; + ret = vk->GetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR(s->hwctx->phys_dev, + &quality_info, + &ctx->quality_props); + if (ret != VK_SUCCESS) + return AVERROR_EXTERNAL; + + /* Printout informative properties */ + av_log(avctx, AV_LOG_VERBOSE, "Encoder capabilities for %s profile \"%s\":\n", + avcodec_get_name(avctx->codec_id), + avcodec_profile_name(avctx->codec_id, avctx->profile)); + av_log(avctx, AV_LOG_VERBOSE, " Width: from %i to %i\n", + ctx->caps.minCodedExtent.width, ctx->caps.maxCodedExtent.width); + av_log(avctx, AV_LOG_VERBOSE, " Height: from %i to %i\n", + ctx->caps.minCodedExtent.height, ctx->caps.maxCodedExtent.height); + av_log(avctx, AV_LOG_VERBOSE, " Width alignment: %i\n", + ctx->caps.pictureAccessGranularity.width); + av_log(avctx, AV_LOG_VERBOSE, " Height alignment: %i\n", + ctx->caps.pictureAccessGranularity.height); + av_log(avctx, AV_LOG_VERBOSE, " Bitstream offset alignment: %"PRIu64"\n", + ctx->caps.minBitstreamBufferOffsetAlignment); + av_log(avctx, AV_LOG_VERBOSE, " Bitstream size alignment: %"PRIu64"\n", + ctx->caps.minBitstreamBufferSizeAlignment); + av_log(avctx, AV_LOG_VERBOSE, " Maximum references: %u\n", + ctx->caps.maxDpbSlots); + av_log(avctx, AV_LOG_VERBOSE, " Maximum active references: %u\n", + ctx->caps.maxActiveReferencePictures); + av_log(avctx, AV_LOG_VERBOSE, " Codec header version: %i.%i.%i (driver), %i.%i.%i (compiled)\n", + CODEC_VER(ctx->caps.stdHeaderVersion.specVersion), + CODEC_VER(vk_desc->ext_props.specVersion)); + av_log(avctx, AV_LOG_VERBOSE, " Encoder max quality: %i\n", + ctx->enc_caps.maxQualityLevels); + av_log(avctx, AV_LOG_VERBOSE, " Encoder image width alignment: %i\n", + ctx->enc_caps.encodeInputPictureGranularity.width); + av_log(avctx, AV_LOG_VERBOSE, " Encoder image height alignment: %i\n", + ctx->enc_caps.encodeInputPictureGranularity.height); + av_log(avctx, AV_LOG_VERBOSE, " Capability flags:%s%s%s\n", + ctx->caps.flags ? "" : + " none", + ctx->caps.flags & VK_VIDEO_CAPABILITY_PROTECTED_CONTENT_BIT_KHR ? + " protected" : "", + ctx->caps.flags & VK_VIDEO_CAPABILITY_SEPARATE_REFERENCE_IMAGES_BIT_KHR ? + " separate_references" : ""); + + /* Setup width/height alignment */ + base_ctx->surface_width = avctx->coded_width = + FFALIGN(avctx->width, ctx->enc_caps.encodeInputPictureGranularity.width); + base_ctx->surface_height = avctx->coded_height = + FFALIGN(avctx->height, ctx->enc_caps.encodeInputPictureGranularity.height); + + /* Check if encoding is possible with the given parameters */ + if (avctx->coded_width < ctx->caps.minCodedExtent.width || + avctx->coded_height < ctx->caps.minCodedExtent.height || + avctx->coded_width > ctx->caps.maxCodedExtent.width || + avctx->coded_height > ctx->caps.maxCodedExtent.height) { + av_log(avctx, AV_LOG_ERROR, "Input of %ix%i too large for encoder limits: %ix%i max\n", + avctx->coded_width, avctx->coded_height, + ctx->caps.minCodedExtent.width, ctx->caps.minCodedExtent.height); + return AVERROR(EINVAL); + } + + fmt_info.imageUsage = VK_IMAGE_USAGE_VIDEO_ENCODE_DPB_BIT_KHR | + VK_IMAGE_USAGE_VIDEO_ENCODE_DST_BIT_KHR; + + ctx->common.layered_dpb = !(ctx->caps.flags & VK_VIDEO_CAPABILITY_SEPARATE_REFERENCE_IMAGES_BIT_KHR); + + /* Get the supported image formats */ + ret = vk->GetPhysicalDeviceVideoFormatPropertiesKHR(s->hwctx->phys_dev, + &fmt_info, + &nb_out_fmts, NULL); + if (ret == VK_ERROR_FORMAT_NOT_SUPPORTED || + (!nb_out_fmts && ret == VK_SUCCESS)) { + return AVERROR(EINVAL); + } else if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Unable to get Vulkan format properties: %s!\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + ret_info = av_mallocz(sizeof(*ret_info)*nb_out_fmts); + if (!ret_info) + return AVERROR(ENOMEM); + + for (int i = 0; i < nb_out_fmts; i++) + ret_info[i].sType = VK_STRUCTURE_TYPE_VIDEO_FORMAT_PROPERTIES_KHR; + + ret = vk->GetPhysicalDeviceVideoFormatPropertiesKHR(s->hwctx->phys_dev, + &fmt_info, + &nb_out_fmts, ret_info); + if (ret == VK_ERROR_FORMAT_NOT_SUPPORTED || + (!nb_out_fmts && ret == VK_SUCCESS)) { + av_free(ret_info); + return AVERROR(EINVAL); + } else if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Unable to get Vulkan format properties: %s!\n", + ff_vk_ret2str(ret)); + av_free(ret_info); + return AVERROR_EXTERNAL; + } + + av_log(avctx, AV_LOG_VERBOSE, "Supported input formats:\n"); + for (i = 0; i < nb_out_fmts; i++) + av_log(avctx, AV_LOG_VERBOSE, " %i: %i\n", i, ret_info[i].format); + + for (i = 0; i < nb_out_fmts; i++) { + if (ff_vk_pix_fmt_from_vkfmt(ret_info[i].format) == s->frames->sw_format) { + ctx->pic_format = ret_info[i].format; + break; + } + } + + av_free(ret_info); + + if (i == nb_out_fmts) { + av_log(avctx, AV_LOG_ERROR, "Pixel format %s of input frames not supported!\n", + av_get_pix_fmt_name(s->frames->sw_format)); + return AVERROR(EINVAL); + } + + /* Create session */ + session_create.pVideoProfile = &ctx->profile; + session_create.flags = 0x0; + session_create.queueFamilyIndex = ctx->qf_enc.queue_family; + session_create.maxCodedExtent = ctx->caps.maxCodedExtent; + session_create.maxDpbSlots = ctx->caps.maxDpbSlots; + session_create.maxActiveReferencePictures = ctx->caps.maxActiveReferencePictures; + session_create.pictureFormat = ctx->pic_format; + session_create.referencePictureFormat = session_create.pictureFormat; + session_create.pStdHeaderVersion = &vk_desc->ext_props; + + err = ff_vk_video_common_init(avctx, s, &ctx->common, &session_create); + if (err < 0) + return err; + + err = ff_hw_base_encode_init(avctx, &ctx->base); + if (err < 0) + return err; + + err = vulkan_encode_create_dpb(avctx, ctx); + if (err < 0) + return err; + + base_ctx->async_encode = 1; + base_ctx->encode_fifo = av_fifo_alloc2(base_ctx->async_depth, + sizeof(FFVulkanEncodePicture *), 0); + if (!base_ctx->encode_fifo) + return AVERROR(ENOMEM); + + return 0; +} diff --git a/libavcodec/vulkan_encode.h b/libavcodec/vulkan_encode.h new file mode 100644 index 0000000000..995befeca2 --- /dev/null +++ b/libavcodec/vulkan_encode.h @@ -0,0 +1,243 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_VULKAN_ENCODE_H +#define AVCODEC_VULKAN_ENCODE_H + +#include "codec_id.h" +#include "internal.h" + +#include "encode.h" +#include "hwconfig.h" + +#include "vulkan_video.h" +#include "hw_base_encode.h" + +typedef struct FFVulkanEncodeDescriptor { + enum AVCodecID codec_id; + FFVulkanExtensions encode_extension; + VkVideoCodecOperationFlagBitsKHR encode_op; + + VkExtensionProperties ext_props; +} FFVulkanEncodeDescriptor; + +typedef struct FFVulkanEncodePicture { + FFHWBaseEncodePicture base; + VkVideoPictureResourceInfoKHR dpb_res; + VkVideoReferenceSlotInfoKHR dpb_slot; + + struct { + VkImageView view; + VkImageAspectFlags aspect; + } in; + + struct { + VkImageView view; + VkImageAspectFlags aspect; + } dpb; + + void *codec_layer; + void *codec_rc_layer; + + FFVkExecContext *exec; + AVBufferRef *pkt_buf; + int slices_offset; +} FFVulkanEncodePicture; + +/** + * Callback for writing stream-level headers. + */ +typedef int (*vkenc_cb_write_stream_headers)(AVCodecContext *avctx, + uint8_t *data, size_t *data_len); + +/** + * Callback for initializing codec-specific picture headers. + */ +typedef int (*vkenc_cb_init_pic_headers)(AVCodecContext *avctx, + FFVulkanEncodePicture *pic); + +/** + * Callback for writing alignment data. + * Align is the value to align offset to. + */ +typedef int (*vkenc_cb_write_filler)(AVCodecContext *avctx, uint32_t filler, + uint8_t *data, size_t *data_len); + +/** + * Callback for writing any extra units requested. data_len must be set + * to the available size, and its value will be overwritten by the #bytes written + * to the output buffer. + */ +typedef int (*vkenc_cb_write_extra_headers)(AVCodecContext *avctx, + FFVulkanEncodePicture *pic, + uint8_t *data, size_t *data_len); + +typedef struct FFVulkanCodec { + /** + * Codec feature flags. + */ + int flags; +/* Codec output packet without timestamp delay, which means the + * output packet has same PTS and DTS. For AV1. */ +#define VK_ENC_FLAG_NO_DELAY 1 << 6 + + /** + * Size of the codec-specific picture struct. + */ + size_t picture_priv_data_size; + + /** + * Size of the filler header. + */ + size_t filler_header_size; + + /** + * Initialize codec-specific structs in a Vulkan profile. + */ + int (*init_profile)(AVCodecContext *avctx, VkVideoProfileInfoKHR *profile, + void *pnext); + + /** + * Initialize codec-specific rate control structures for a picture. + */ + int (*init_pic_rc)(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + VkVideoEncodeRateControlInfoKHR *rc_info, + VkVideoEncodeRateControlLayerInfoKHR *rc_layer); + + /** + * Initialize codec-specific picture parameters. + */ + int (*init_pic_params)(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + VkVideoEncodeInfoKHR *encode_info); + + /** + * Callback for writing stream headers. + */ + int (*write_sequence_headers)(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic, + uint8_t *data, size_t *data_len); + + /** + * Callback for writing alignment data. + */ + int (*write_filler)(AVCodecContext *avctx, uint32_t filler, + uint8_t *data, size_t *data_len); + + /** + * Callback for writing any extra units requested. data_len must be set + * to the available size, and its value will be overwritten by the #bytes written + * to the output buffer. + */ + int (*write_extra_headers)(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + uint8_t *data, size_t *data_len); +} FFVulkanCodec; + +typedef struct FFVkEncodeCommonOptions { + int qp; + int quality; + int profile; + int level; + int async_depth; + VkVideoEncodeUsageFlagBitsKHR usage; + VkVideoEncodeContentFlagBitsKHR content; + VkVideoEncodeTuningModeKHR tune; + + VkVideoEncodeRateControlModeFlagBitsKHR rc_mode; +#define FF_VK_RC_MODE_AUTO 0xFFFFFFFF +} FFVkEncodeCommonOptions; + +typedef struct FFVulkanEncodeContext { + FFVulkanContext s; + FFVkVideoCommon common; + FFHWBaseEncodeContext base; + const FFVulkanCodec *codec; + + /* Session parameters object, initialized by each codec independently + * and set here. */ + VkVideoSessionParametersKHR session_params; + + AVBufferPool *buf_pool; + + VkFormat pic_format; + + FFVkEncodeCommonOptions opts; + + VkVideoProfileInfoKHR profile; + VkVideoProfileListInfoKHR profile_list; + VkVideoCapabilitiesKHR caps; + VkVideoEncodeQualityLevelPropertiesKHR quality_props; + VkVideoEncodeCapabilitiesKHR enc_caps; + VkVideoEncodeUsageInfoKHR usage_info; + + FFVkQueueFamilyCtx qf_enc; + FFVkExecPool enc_pool; + + uint32_t slots[32]; +} FFVulkanEncodeContext; + +#define VULKAN_ENCODE_COMMON_OPTIONS \ + { "qp", "Use an explicit constant quantizer for the whole stream", OFFSET(common.opts.qp), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, 255, FLAGS }, \ + { "quality", "Set encode quality (trades off against speed, higher is faster)", OFFSET(common.opts.quality), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, FLAGS }, \ + { "rc_mode", "Select rate control type", OFFSET(common.opts.rc_mode), AV_OPT_TYPE_INT, { .i64 = FF_VK_RC_MODE_AUTO }, 0, FF_VK_RC_MODE_AUTO, FLAGS, "rc_mode" }, \ + { "auto", "Choose mode automatically based on parameters", 0, AV_OPT_TYPE_CONST, { .i64 = FF_VK_RC_MODE_AUTO }, INT_MIN, INT_MAX, FLAGS, "rc_mode" }, \ + { "driver", "Driver-specific rate control", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DEFAULT_KHR }, INT_MIN, INT_MAX, FLAGS, "rc_mode" }, \ + { "cqp", "Constant quantizer mode", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "rc_mode" }, \ + { "cbr", "Constant bitrate mode", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "rc_mode" }, \ + { "vbr", "Variable bitrate mode", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "rc_mode" }, \ + { "tune", "Select tuning type", OFFSET(common.opts.tune), AV_OPT_TYPE_INT, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_DEFAULT_KHR }, 0, INT_MAX, FLAGS, "tune" }, \ + { "default", "Default tuning", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_DEFAULT_KHR }, INT_MIN, INT_MAX, FLAGS, "tune" }, \ + { "hq", "High quality tuning", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_HIGH_QUALITY_KHR }, INT_MIN, INT_MAX, FLAGS, "tune" }, \ + { "ll", "Low-latency tuning", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_LOW_LATENCY_KHR }, INT_MIN, INT_MAX, FLAGS, "tune" }, \ + { "ull", "Ultra low-latency tuning", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_ULTRA_LOW_LATENCY_KHR }, INT_MIN, INT_MAX, FLAGS, "tune" }, \ + { "lossless", "Lossless mode tuning", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_TUNING_MODE_LOSSLESS_KHR }, INT_MIN, INT_MAX, FLAGS, "tune" }, \ + { "usage", "Select usage type", OFFSET(common.opts.usage), AV_OPT_TYPE_FLAGS, { .i64 = VK_VIDEO_ENCODE_USAGE_DEFAULT_KHR }, 0, INT_MAX, FLAGS, "usage" }, \ + { "default", "Default optimizations", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_USAGE_DEFAULT_KHR }, INT_MIN, INT_MAX, FLAGS, "usage" }, \ + { "transcode", "Optimize for transcoding", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_USAGE_TRANSCODING_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "usage" }, \ + { "stream", "Optimize for streaming", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_USAGE_STREAMING_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "usage" }, \ + { "record", "Optimize for offline recording", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_USAGE_RECORDING_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "usage" }, \ + { "conference", "Optimize for teleconferencing", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_USAGE_CONFERENCING_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "usage" }, \ + { "content", "Select content type", OFFSET(common.opts.content), AV_OPT_TYPE_FLAGS, { .i64 = VK_VIDEO_ENCODE_CONTENT_DEFAULT_KHR }, 0, INT_MAX, FLAGS, "content" }, \ + { "default", "Default content", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_CONTENT_DEFAULT_KHR }, INT_MIN, INT_MAX, FLAGS, "content" }, \ + { "camera", "Camera footage", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_CONTENT_CAMERA_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "content" }, \ + { "desktop", "Screen recording", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_CONTENT_DESKTOP_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "content" }, \ + { "rendered", "Game or 3D content", 0, AV_OPT_TYPE_CONST, { .i64 = VK_VIDEO_ENCODE_CONTENT_RENDERED_BIT_KHR }, INT_MIN, INT_MAX, FLAGS, "content" } + +/** + * Initialize encoder. + */ +av_cold int ff_vulkan_encode_init(AVCodecContext *avctx, FFVulkanEncodeContext *ctx, + const FFVulkanEncodeDescriptor *vk_desc, + const FFVulkanCodec *codec, + void *codec_caps, void *quality_pnext); + +/** + * Encode. + */ +int ff_vulkan_encode_receive_packet(AVCodecContext *avctx, AVPacket *pkt); + +/** + * Uninitialize encoder. + */ +void ff_vulkan_encode_uninit(FFVulkanEncodeContext *ctx); + +/** + * Paperwork. + */ +extern const AVCodecHWConfigInternal *const ff_vulkan_encode_hw_configs[]; + +#endif /* AVCODEC_VULKAN_ENCODE_H */ From patchwork Mon Sep 9 10:37:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lynne X-Patchwork-Id: 51474 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a59:9c4f:0:b0:48e:c0f8:d0de with SMTP id w15csp1544337vqu; Mon, 9 Sep 2024 03:39:14 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCUrpKLjAesmlstOi4XClCfgO4tbe8wQQcmqCVrwzbd0QOPj5fhIqvp6TFUEbqgQy2nSquuSwVVOkEFNcn7uLveT@gmail.com X-Google-Smtp-Source: AGHT+IEWq46+MEKamVCMxyBNmIfPP4XhEWwfI6M6IxuQEKcpzTwj4WWTflP1UDEotE8yYmn9j5jX X-Received: by 2002:a05:6512:118d:b0:52e:ccf5:7c3e with SMTP id 2adb3069b0e04-536588226e8mr3480680e87.7.1725878353831; Mon, 09 Sep 2024 03:39:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1725878353; cv=none; d=google.com; s=arc-20240605; b=KeD9U+D1B8KyjeLYK3ClQvIPQZeMM9e36ysq6b5WO23bLJT5nzYoWuzW4oN0MnO2X3 bNW8k9Fy2NUKIm9C099EokeRswqpI6XQkxYHuDwYjQWZqSqHneOV+v81tGTON5yIWQs8 lWcTKg1KcIIO9Q8l4aMZS5sYpPSl0Rpz6UPhwICtqKlye7eCYW7QhbSnDoT3G5IqMEFy dzGzxwS8P56tlvR5QnQylNY5c/edbNB9/ejkWAgw9JHEsr/ovBu7vhpwj6/ZUDcoGToB t9cm/ybxrDZc+k6HScDa2J4LcFqQibSTp3NV19g67XhlY3SJd5Fjmz57lAPTvlgiJqZG m3Og== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=sender:errors-to:content-transfer-encoding:cc:reply-to:from :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:delivered-to; bh=Ej6NTDxaiuoGSfN9ww18ojQgLXLeY0RS4HVlGbkdLcc=; fh=nenT92/WZoU6unXd3J6UhGUdod4piddKfVtctNBOh6k=; b=ckEO1v4Y7aYsW25EaY5pIoHdOcffPBFT6BDUyC3JE2SR02E+Q0iI1DGdeMqY6n12t4 n/fHkqE2aTQGwZPZotaNlU9XfwSGwCVGM0R9wG/VzzZNjIla/qUr3MyQhoBScQgoLNW+ pNxQswpS+BXkFWRGWZdw/5xSPx5X2UVyZHjQmwHQclLE6QAqIa8pqYD23tCFcNncqI01 ZWnI+/J2ha7lh1bc5fZKtmKI3K1J8VmaDcKi+ZC0kD4C9DknZ/eUWMpO2YznO3pzkFU7 vhZE3Cu0tc9qJBCDrbCkjxuUt09+DpdfKeoC2oEC2TDY5fPKngwDgX26vC166Fbcq4// 2r4g==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 2adb3069b0e04-5365f866371si1588247e87.63.2024.09.09.03.39.13; Mon, 09 Sep 2024 03:39:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id C996868DCB0; Mon, 9 Sep 2024 13:39:09 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from vidala.lynne.ee (vidala.pars.ee [116.203.72.101]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 8A4A568DF0B for ; Mon, 9 Sep 2024 13:39:02 +0300 (EEST) To: ffmpeg-devel@ffmpeg.org Date: Mon, 9 Sep 2024 12:37:52 +0200 Message-ID: <20240909103759.371919-2-dev@lynne.ee> X-Mailer: git-send-email 2.45.2.753.g447d99e1c3b In-Reply-To: <20240909103759.371919-1-dev@lynne.ee> References: <20240909103759.371919-1-dev@lynne.ee> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] lavc: add h264_vulkan hardware encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Lynne via ffmpeg-devel From: Lynne Reply-To: FFmpeg development discussions and patches Cc: Lynne Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: DXSzXVKR6Xgk This commit adds the first Vulkan hardware encoder. Currently, P, and **B**-frames are supported. This marks the first implementation to support both. The encoder has feature-parity with VAAPI. --- configure | 1 + libavcodec/Makefile | 3 + libavcodec/allcodecs.c | 1 + libavcodec/vulkan_encode_h264.c | 1687 +++++++++++++++++++++++++++++++ 4 files changed, 1692 insertions(+) create mode 100644 libavcodec/vulkan_encode_h264.c diff --git a/configure b/configure index 6cfb736a86..ebb8be73ad 100755 --- a/configure +++ b/configure @@ -3366,6 +3366,7 @@ h264_qsv_encoder_select="atsc_a53 qsvenc" h264_rkmpp_decoder_deps="rkmpp" h264_rkmpp_decoder_select="h264_mp4toannexb_bsf" h264_vaapi_encoder_select="atsc_a53 cbs_h264 vaapi_encode" +h264_vulkan_encoder_select="cbs_h264 vulkan_encode" h264_v4l2m2m_decoder_deps="v4l2_m2m h264_v4l2_m2m" h264_v4l2m2m_decoder_select="h264_mp4toannexb_bsf" h264_v4l2m2m_encoder_deps="v4l2_m2m h264_v4l2_m2m" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index ff6a3c4efc..d4eefa28a2 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -429,6 +429,9 @@ OBJS-$(CONFIG_H264_VAAPI_ENCODER) += vaapi_encode_h264.o h264_levels.o \ OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o OBJS-$(CONFIG_H264_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_H264_V4L2M2M_ENCODER) += v4l2_m2m_enc.o +OBJS-$(CONFIG_H264_VULKAN_ENCODER) += vulkan_encode.o vulkan_encode_h264.o \ + hw_base_encode.o hw_base_encode_h264.o \ + h264_levels.o h2645data.o OBJS-$(CONFIG_HAP_DECODER) += hapdec.o hap.o OBJS-$(CONFIG_HAP_ENCODER) += hapenc.o hap.o OBJS-$(CONFIG_HCA_DECODER) += hcadec.o diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index d773ac36c2..cfd929b81f 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -850,6 +850,7 @@ extern const FFCodec ff_h264_qsv_encoder; extern const FFCodec ff_h264_v4l2m2m_encoder; extern const FFCodec ff_h264_vaapi_encoder; extern const FFCodec ff_h264_videotoolbox_encoder; +extern const FFCodec ff_h264_vulkan_encoder; extern const FFCodec ff_hevc_amf_encoder; extern const FFCodec ff_hevc_cuvid_decoder; extern const FFCodec ff_hevc_d3d12va_encoder; diff --git a/libavcodec/vulkan_encode_h264.c b/libavcodec/vulkan_encode_h264.c new file mode 100644 index 0000000000..529ac8aed2 --- /dev/null +++ b/libavcodec/vulkan_encode_h264.c @@ -0,0 +1,1687 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/opt.h" +#include "libavutil/mem.h" + +#include "cbs.h" +#include "cbs_h264.h" +#include "atsc_a53.h" + +#include "h264_levels.h" +#include "h2645data.h" +#include "codec_internal.h" +#include "version.h" +#include "hw_base_encode_h264.h" + +#include "vulkan_encode.h" + +enum UnitElems { + UNIT_AUD = 1 << 0, + UNIT_SEI_TIMING = 1 << 1, + UNIT_SEI_IDENTIFIER = 1 << 2, + UNIT_SEI_RECOVERY = 1 << 3, + UNIT_SEI_A53_CC = 1 << 4, +}; + +const FFVulkanEncodeDescriptor ff_vk_enc_h264_desc = { + .codec_id = AV_CODEC_ID_H264, + .encode_extension = FF_VK_EXT_VIDEO_ENCODE_H264, + .encode_op = VK_VIDEO_CODEC_OPERATION_ENCODE_H264_BIT_KHR, + .ext_props = { + .extensionName = VK_STD_VULKAN_VIDEO_CODEC_H264_ENCODE_EXTENSION_NAME, + .specVersion = VK_STD_VULKAN_VIDEO_CODEC_H264_ENCODE_SPEC_VERSION, + }, +}; + +/* Random (version 4) ISO 11578 UUID. */ +static const uint8_t vulkan_encode_h264_sei_identifier_uuid[16] = { + 0x03, 0xfd, 0xf2, 0x0a, 0x5d, 0x4c, 0x05, 0x48, + 0x20, 0x98, 0xca, 0x6b, 0x0c, 0x95, 0x30, 0x1c, +}; + +typedef struct VulkanEncodeH264Picture { + int frame_num; + int64_t last_idr_frame; + uint16_t idr_pic_id; + int primary_pic_type; + int slice_type; + int pic_order_cnt; + + enum UnitElems units_needed; + + VkVideoEncodeH264RateControlInfoKHR vkrc_info; + VkVideoEncodeH264RateControlLayerInfoKHR vkrc_layer_info; + + StdVideoEncodeH264WeightTable slice_wt; + StdVideoEncodeH264SliceHeader slice_hdr; + VkVideoEncodeH264NaluSliceInfoKHR vkslice; + + StdVideoEncodeH264PictureInfo h264pic_info; + VkVideoEncodeH264PictureInfoKHR vkh264pic_info; + + StdVideoEncodeH264ReferenceInfo h264dpb_info; + VkVideoEncodeH264DpbSlotInfoKHR vkh264dpb_info; + + StdVideoEncodeH264RefListModEntry mods[MAX_REFERENCE_LIST_NUM][H264_MAX_RPLM_COUNT]; + StdVideoEncodeH264RefPicMarkingEntry mmco[H264_MAX_RPLM_COUNT]; + StdVideoEncodeH264ReferenceListsInfo ref_list_info; +} VulkanEncodeH264Picture; + +typedef struct VulkanEncodeH264Context { + FFVulkanEncodeContext common; + + FFHWBaseEncodeH264 units; + FFHWBaseEncodeH264Opts unit_opts; + + enum UnitElems unit_elems; + + uint8_t fixed_qp_idr; + uint8_t fixed_qp_p; + uint8_t fixed_qp_b; + + VkVideoEncodeH264ProfileInfoKHR profile; + + VkVideoEncodeH264CapabilitiesKHR caps; + VkVideoEncodeH264QualityLevelPropertiesKHR quality_props; + + CodedBitstreamContext *cbs; + CodedBitstreamFragment current_access_unit; + + H264RawAUD raw_aud; + + SEIRawUserDataUnregistered sei_identifier; + H264RawSEIPicTiming sei_pic_timing; + H264RawSEIRecoveryPoint sei_recovery_point; + SEIRawUserDataRegistered sei_a53cc; + void *sei_a53cc_data; + char *sei_identifier_string; +} VulkanEncodeH264Context; + +static int init_pic_rc(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + VkVideoEncodeRateControlInfoKHR *rc_info, + VkVideoEncodeRateControlLayerInfoKHR *rc_layer) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + VulkanEncodeH264Picture *hp = pic->codec_priv; + + hp->vkrc_info = (VkVideoEncodeH264RateControlInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_RATE_CONTROL_INFO_KHR, + .flags = VK_VIDEO_ENCODE_H264_RATE_CONTROL_REFERENCE_PATTERN_FLAT_BIT_KHR | + VK_VIDEO_ENCODE_H264_RATE_CONTROL_REGULAR_GOP_BIT_KHR, + .idrPeriod = ctx->base.gop_size, + .gopFrameCount = ctx->base.gop_size, + .consecutiveBFrameCount = FFMAX(ctx->base.b_per_p - 1, 0), + .temporalLayerCount = 0, + }; + + rc_info->pNext = &hp->vkrc_info; + rc_info->virtualBufferSizeInMs = enc->unit_opts.hrd_buffer_size; + rc_info->initialVirtualBufferSizeInMs = enc->unit_opts.initial_buffer_fullness; + + if (rc_info->rateControlMode > VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR) { + hp->vkrc_layer_info = (VkVideoEncodeH264RateControlLayerInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_RATE_CONTROL_LAYER_INFO_KHR, + + .useMinQp = avctx->qmin > 0, + .minQp.qpI = avctx->qmin > 0 ? avctx->qmin : 0, + .minQp.qpP = avctx->qmin > 0 ? avctx->qmin : 0, + .minQp.qpB = avctx->qmin > 0 ? avctx->qmin : 0, + + .useMaxQp = avctx->qmax > 0, + .maxQp.qpI = avctx->qmax > 0 ? avctx->qmax : 0, + .maxQp.qpP = avctx->qmax > 0 ? avctx->qmax : 0, + .maxQp.qpB = avctx->qmax > 0 ? avctx->qmax : 0, + + .useMaxFrameSize = 0, + }; + rc_layer->pNext = &hp->vkrc_layer_info; + hp->vkrc_info.temporalLayerCount = 1; + } + + return 0; +} + +static int vk_enc_h264_update_pic_info(AVCodecContext *avctx, + FFHWBaseEncodePicture *pic) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + VulkanEncodeH264Picture *hp = pic->codec_priv; + FFHWBaseEncodePicture *prev = pic->prev; + VulkanEncodeH264Picture *hprev = prev ? prev->codec_priv : NULL; + + if (pic->type == FF_HW_PICTURE_TYPE_IDR) { + av_assert0(pic->display_order == pic->encode_order); + + hp->frame_num = 0; + hp->last_idr_frame = pic->display_order; + hp->idr_pic_id = hprev ? hprev->idr_pic_id + 1 : 0; + + hp->primary_pic_type = 0; + hp->slice_type = STD_VIDEO_H264_SLICE_TYPE_I; + } else { + av_assert0(prev); + + hp->frame_num = hprev->frame_num + prev->is_reference; + + hp->last_idr_frame = hprev->last_idr_frame; + hp->idr_pic_id = hprev->idr_pic_id; + + if (pic->type == FF_HW_PICTURE_TYPE_I) { + hp->slice_type = STD_VIDEO_H264_SLICE_TYPE_I; + hp->primary_pic_type = 0; + } else if (pic->type == FF_HW_PICTURE_TYPE_P) { + hp->slice_type = STD_VIDEO_H264_SLICE_TYPE_P; + hp->primary_pic_type = 1; + } else { + hp->slice_type = STD_VIDEO_H264_SLICE_TYPE_B; + hp->primary_pic_type = 2; + } + } + + hp->pic_order_cnt = pic->display_order - hp->last_idr_frame; + if (enc->units.raw_sps.pic_order_cnt_type == 2) + hp->pic_order_cnt *= 2; + + hp->units_needed = 0; + + if (enc->unit_elems & UNIT_SEI_IDENTIFIER && pic->encode_order == 0) + hp->units_needed |= UNIT_SEI_IDENTIFIER; + + if (enc->unit_elems & UNIT_SEI_TIMING) { + enc->sei_pic_timing = (H264RawSEIPicTiming) { + .cpb_removal_delay = 2 * (pic->encode_order - hp->last_idr_frame), + .dpb_output_delay = 2 * (pic->display_order - pic->encode_order + ctx->base.max_b_depth), + }; + + hp->units_needed |= UNIT_SEI_TIMING; + } + + if (enc->unit_elems & UNIT_SEI_RECOVERY && pic->type == FF_HW_PICTURE_TYPE_I) { + enc->sei_recovery_point = (H264RawSEIRecoveryPoint) { + .recovery_frame_cnt = 0, + .exact_match_flag = 1, + .broken_link_flag = ctx->base.b_per_p > 0, + }; + + hp->units_needed |= UNIT_SEI_RECOVERY; + } + + if (enc->unit_elems & UNIT_SEI_A53_CC) { + int err; + size_t sei_a53cc_len; + av_freep(&enc->sei_a53cc_data); + err = ff_alloc_a53_sei(pic->input_image, 0, &enc->sei_a53cc_data, &sei_a53cc_len); + if (err < 0) + return err; + if (enc->sei_a53cc_data != NULL) { + enc->sei_a53cc.itu_t_t35_country_code = 181; + enc->sei_a53cc.data = (uint8_t *)enc->sei_a53cc_data + 1; + enc->sei_a53cc.data_length = sei_a53cc_len - 1; + + hp->units_needed |= UNIT_SEI_A53_CC; + } + } + + return 0; +} + +static void setup_slices(AVCodecContext *avctx, + FFHWBaseEncodePicture *pic) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + VulkanEncodeH264Picture *hp = pic->codec_priv; + + hp->slice_wt = (StdVideoEncodeH264WeightTable) { + .flags = (StdVideoEncodeH264WeightTableFlags) { + .luma_weight_l0_flag = 0, + .chroma_weight_l0_flag = 0, + .luma_weight_l1_flag = 0, + .chroma_weight_l1_flag = 0, + }, + .luma_log2_weight_denom = 0, + .chroma_log2_weight_denom = 0, + .luma_weight_l0 = { 0 }, + .luma_offset_l0 = { 0 }, + .chroma_weight_l0 = { { 0 } }, + .chroma_offset_l0 = { { 0 } }, + .luma_weight_l1 = { 0 }, + .luma_offset_l1 = { 0 }, + .chroma_weight_l1 = { { 0 } }, + .chroma_offset_l1 = { { 0 } }, + }; + + hp->slice_hdr = (StdVideoEncodeH264SliceHeader) { + .flags = (StdVideoEncodeH264SliceHeaderFlags) { + .direct_spatial_mv_pred_flag = 1, + /* The vk_samples code does this */ + .num_ref_idx_active_override_flag = + ((enc->units.raw_pps.num_ref_idx_l0_default_active_minus1) && + (pic->type == FF_HW_PICTURE_TYPE_B)) ? 1 : 0, + }, + .first_mb_in_slice = 1, + .slice_type = hp->slice_type, + .slice_alpha_c0_offset_div2 = 0, + .slice_beta_offset_div2 = 0, + .slice_qp_delta = 0, /* Filled in below */ + /* Reserved */ + .cabac_init_idc = 0, + .disable_deblocking_filter_idc = 0, + .pWeightTable = NULL, // &hp->slice_wt, + }; + + hp->vkslice = (VkVideoEncodeH264NaluSliceInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_NALU_SLICE_INFO_KHR, + .pNext = NULL, + .constantQp = pic->type == FF_HW_PICTURE_TYPE_B ? enc->fixed_qp_b : + pic->type == FF_HW_PICTURE_TYPE_P ? enc->fixed_qp_p : + enc->fixed_qp_idr, + .pStdSliceHeader = &hp->slice_hdr, + }; + + hp->slice_hdr.slice_qp_delta = hp->vkslice.constantQp - + (enc->units.raw_pps.pic_init_qp_minus26 + 26); + + if (enc->common.opts.rc_mode != VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR) + hp->vkslice.constantQp = 0; + + hp->vkh264pic_info.pNaluSliceEntries = &hp->vkslice; + hp->vkh264pic_info.naluSliceEntryCount = 1; +} + +static void vk_enc_h264_default_ref_pic_list(AVCodecContext *avctx, + FFHWBaseEncodePicture *pic, + FFHWBaseEncodePicture **rpl0, + FFHWBaseEncodePicture **rpl1, + int *rpl_size) +{ + FFHWBaseEncodePicture *prev; + VulkanEncodeH264Picture *hp, *hn, *hc; + int i, j, n = 0; + + prev = pic->prev; + av_assert0(prev); + hp = pic->codec_priv; + + for (i = 0; i < pic->prev->nb_dpb_pics; i++) { + hn = prev->dpb[i]->codec_priv; + av_assert0(hn->frame_num < hp->frame_num); + + if (pic->type == FF_HW_PICTURE_TYPE_P) { + for (j = n; j > 0; j--) { + hc = rpl0[j - 1]->codec_priv; + av_assert0(hc->frame_num != hn->frame_num); + if (hc->frame_num > hn->frame_num) + break; + rpl0[j] = rpl0[j - 1]; + } + rpl0[j] = prev->dpb[i]; + + } else if (pic->type == FF_HW_PICTURE_TYPE_B) { + for (j = n; j > 0; j--) { + hc = rpl0[j - 1]->codec_priv; + av_assert0(hc->pic_order_cnt != hp->pic_order_cnt); + if (hc->pic_order_cnt < hp->pic_order_cnt) { + if (hn->pic_order_cnt > hp->pic_order_cnt || + hn->pic_order_cnt < hc->pic_order_cnt) + break; + } else { + if (hn->pic_order_cnt > hc->pic_order_cnt) + break; + } + rpl0[j] = rpl0[j - 1]; + } + rpl0[j] = prev->dpb[i]; + + for (j = n; j > 0; j--) { + hc = rpl1[j - 1]->codec_priv; + av_assert0(hc->pic_order_cnt != hp->pic_order_cnt); + if (hc->pic_order_cnt > hp->pic_order_cnt) { + if (hn->pic_order_cnt < hp->pic_order_cnt || + hn->pic_order_cnt > hc->pic_order_cnt) + break; + } else { + if (hn->pic_order_cnt < hc->pic_order_cnt) + break; + } + rpl1[j] = rpl1[j - 1]; + } + rpl1[j] = prev->dpb[i]; + } + + ++n; + } + + if (pic->type == FF_HW_PICTURE_TYPE_B) { + for (i = 0; i < n; i++) { + if (rpl0[i] != rpl1[i]) + break; + } + if (i == n) + FFSWAP(FFHWBaseEncodePicture *, rpl1[0], rpl1[1]); + } + + if (pic->type == FF_HW_PICTURE_TYPE_P || + pic->type == FF_HW_PICTURE_TYPE_B) { + av_log(avctx, AV_LOG_DEBUG, "Default RefPicList0 for fn=%d/poc=%d:", + hp->frame_num, hp->pic_order_cnt); + for (i = 0; i < n; i++) { + hn = rpl0[i]->codec_priv; + av_log(avctx, AV_LOG_DEBUG, " fn=%d/poc=%d", + hn->frame_num, hn->pic_order_cnt); + } + av_log(avctx, AV_LOG_DEBUG, "\n"); + } + if (pic->type == FF_HW_PICTURE_TYPE_B) { + av_log(avctx, AV_LOG_DEBUG, "Default RefPicList1 for fn=%d/poc=%d:", + hp->frame_num, hp->pic_order_cnt); + for (i = 0; i < n; i++) { + hn = rpl1[i]->codec_priv; + av_log(avctx, AV_LOG_DEBUG, " fn=%d/poc=%d", + hn->frame_num, hn->pic_order_cnt); + } + av_log(avctx, AV_LOG_DEBUG, "\n"); + } + + *rpl_size = n; +} + +static void setup_refs(AVCodecContext *avctx, + FFHWBaseEncodePicture *pic, + VkVideoEncodeInfoKHR *encode_info) +{ + int idx, n, i, j; + VulkanEncodeH264Context *enc = avctx->priv_data; + VulkanEncodeH264Picture *hp = pic->codec_priv; + FFHWBaseEncodePicture *prev = pic->prev; + FFHWBaseEncodePicture *def_l0[MAX_DPB_SIZE], *def_l1[MAX_DPB_SIZE]; + VulkanEncodeH264Picture *href; + + hp->ref_list_info = (StdVideoEncodeH264ReferenceListsInfo) { + .flags = (StdVideoEncodeH264ReferenceListsInfoFlags) { + .ref_pic_list_modification_flag_l0 = 0, + .ref_pic_list_modification_flag_l1 = 0, + /* Reserved */ + }, + /* May be overridden during setup_slices() */ + .num_ref_idx_l0_active_minus1 = pic->nb_refs[0] - 1, + .num_ref_idx_l1_active_minus1 = pic->nb_refs[1] - 1, + /* .RefPicList0 is set in vk_enc_h264_default_ref_pic_list() */ + /* .RefPicList1 is set in vk_enc_h264_default_ref_pic_list() */ + /* Reserved */ + .pRefList0ModOperations = NULL, /* All set below */ + .refList0ModOpCount = 0, + .pRefList1ModOperations = NULL, + .refList1ModOpCount = 0, + .pRefPicMarkingOperations = NULL, + .refPicMarkingOpCount = 0, + }; + + for (i = 0; i < STD_VIDEO_H264_MAX_NUM_LIST_REF; i++) + hp->ref_list_info.RefPicList0[i] = hp->ref_list_info.RefPicList1[i] = -1; + + /* Note: really not sure */ + for (int i = 0; i < pic->nb_refs[0]; i++) { + FFHWBaseEncodePicture *ref = pic->refs[0][i]; + FFVulkanEncodePicture *rvp = ref->priv; + VkVideoReferenceSlotInfoKHR *slot_info; + slot_info = (VkVideoReferenceSlotInfoKHR *)&encode_info->pReferenceSlots[i]; + hp->ref_list_info.RefPicList0[i] = slot_info->slotIndex; + } + + /* Note: really not sure */ + for (int i = 0; i < pic->nb_refs[1]; i++) { + FFHWBaseEncodePicture *ref = pic->refs[1][i]; + FFVulkanEncodePicture *rvp = ref->priv; + VkVideoReferenceSlotInfoKHR *slot_info; + slot_info = (VkVideoReferenceSlotInfoKHR *)&encode_info->pReferenceSlots[pic->nb_refs[0] + i]; + hp->ref_list_info.RefPicList1[i] = slot_info->slotIndex; + } + + hp->h264pic_info.pRefLists = &hp->ref_list_info; + + if (pic->is_reference && pic->type != FF_HW_PICTURE_TYPE_IDR) { + FFHWBaseEncodePicture *discard_list[MAX_DPB_SIZE]; + int discard = 0, keep = 0; + + // Discard everything which is in the DPB of the previous frame but + // not in the DPB of this one. + for (i = 0; i < prev->nb_dpb_pics; i++) { + for (j = 0; j < pic->nb_dpb_pics; j++) { + if (prev->dpb[i] == pic->dpb[j]) + break; + } + if (j == pic->nb_dpb_pics) { + discard_list[discard] = prev->dpb[i]; + ++discard; + } else { + ++keep; + } + } + av_assert0(keep <= enc->units.dpb_frames); + + if (discard == 0) { + hp->h264pic_info.flags.adaptive_ref_pic_marking_mode_flag = 0; + } else { + hp->h264pic_info.flags.adaptive_ref_pic_marking_mode_flag = 1; + for (i = 0; i < discard; i++) { + VulkanEncodeH264Picture *old = discard_list[i]->codec_priv; + av_assert0(old->frame_num < hp->frame_num); + hp->mmco[i] = (StdVideoEncodeH264RefPicMarkingEntry) { + .memory_management_control_operation = 1, + .difference_of_pic_nums_minus1 = hp->frame_num - old->frame_num - 1, + }; + } + hp->ref_list_info.pRefPicMarkingOperations = hp->mmco; + hp->ref_list_info.refPicMarkingOpCount = i - 1; + } + } + + if (pic->type == FF_HW_PICTURE_TYPE_I || pic->type == FF_HW_PICTURE_TYPE_IDR) + return; + + // If the intended references are not the first entries of RefPicListN + // by default, use ref-pic-list-modification to move them there. + vk_enc_h264_default_ref_pic_list(avctx, pic, + def_l0, def_l1, &n); + + if (pic->type == FF_HW_PICTURE_TYPE_P) { + int need_rplm = 0; + for (i = 0; i < pic->nb_refs[0]; i++) { + av_assert0(pic->refs[0][i]); + if (pic->refs[0][i] != (FFHWBaseEncodePicture *)def_l0[i]) + need_rplm = 1; + } + + hp->ref_list_info.flags.ref_pic_list_modification_flag_l0 = need_rplm; + if (need_rplm) { + int pic_num = hp->frame_num; + for (i = 0; i < pic->nb_refs[0]; i++) { + href = pic->refs[0][i]->codec_priv; + av_assert0(href->frame_num != pic_num); + if (href->frame_num < pic_num) { + hp->mods[0][i] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 0, + .abs_diff_pic_num_minus1 = pic_num - href->frame_num - 1, + }; + } else { + hp->mods[0][i] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 1, + .abs_diff_pic_num_minus1 = href->frame_num - pic_num - 1, + }; + } + pic_num = href->frame_num; + } + hp->ref_list_info.pRefList0ModOperations = hp->mods[0]; + hp->ref_list_info.refList0ModOpCount = i - 1; + } + } else { + int need_rplm_l0 = 0, need_rplm_l1 = 0; + int n0 = 0, n1 = 0; + for (i = 0; i < pic->nb_refs[0]; i++) { + av_assert0(pic->refs[0][i]); + href = pic->refs[0][i]->codec_priv; + av_assert0(href->pic_order_cnt < hp->pic_order_cnt); + if (pic->refs[0][i] != (FFHWBaseEncodePicture *)def_l0[n0]) + need_rplm_l0 = 1; + ++n0; + } + + for (int i = 0; i < pic->nb_refs[1]; i++) { + av_assert0(pic->refs[1][i]); + href = pic->refs[1][i]->codec_priv; + av_assert0(href->pic_order_cnt > hp->pic_order_cnt); + if (pic->refs[1][i] != (FFHWBaseEncodePicture *)def_l1[n1]) + need_rplm_l1 = 1; + ++n1; + } + + hp->ref_list_info.flags.ref_pic_list_modification_flag_l0 = need_rplm_l0; + if (need_rplm_l0) { + int pic_num = hp->frame_num; + for (i = j = 0; i < pic->nb_refs[0]; i++) { + href = pic->refs[0][i]->codec_priv; + av_assert0(href->frame_num != pic_num); + if (href->frame_num < pic_num) { + hp->mods[0][j] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 0, + .abs_diff_pic_num_minus1 = pic_num - href->frame_num - 1, + }; + } else { + hp->mods[0][j] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 1, + .abs_diff_pic_num_minus1 = href->frame_num - pic_num - 1, + }; + } + pic_num = href->frame_num; + ++j; + } + hp->ref_list_info.pRefList0ModOperations = hp->mods[0]; + hp->ref_list_info.refList0ModOpCount = j - 1; + } + + hp->ref_list_info.flags.ref_pic_list_modification_flag_l1 = need_rplm_l1; + if (need_rplm_l1) { + int pic_num = hp->frame_num; + for (i = j = 0; i < pic->nb_refs[1]; i++) { + href = pic->refs[1][i]->codec_priv; + av_assert0(href->frame_num != pic_num); + if (href->frame_num < pic_num) { + hp->mods[1][j] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 0, + .abs_diff_pic_num_minus1 = pic_num - href->frame_num - 1, + }; + } else { + hp->mods[1][j] = (StdVideoEncodeH264RefListModEntry) { + .modification_of_pic_nums_idc = 1, + .abs_diff_pic_num_minus1 = href->frame_num - pic_num - 1, + }; + } + pic_num = href->frame_num; + ++j; + } + hp->ref_list_info.pRefList1ModOperations = hp->mods[1]; + hp->ref_list_info.refList1ModOpCount = j - 1; + } + } +} + +static int init_pic_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, + VkVideoEncodeInfoKHR *encode_info) +{ + int err; + FFVulkanEncodePicture *vp = pic->priv; + VulkanEncodeH264Picture *hp = pic->codec_priv; + VkVideoReferenceSlotInfoKHR *ref_slot; + + err = vk_enc_h264_update_pic_info(avctx, pic); + if (err < 0) + return err; + + hp->vkh264pic_info = (VkVideoEncodeH264PictureInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_PICTURE_INFO_KHR, + .pNext = NULL, + .pNaluSliceEntries = NULL, // Filled in during setup_slices() + .naluSliceEntryCount = 0, // Filled in during setup_slices() + .pStdPictureInfo = &hp->h264pic_info, + }; + + hp->h264pic_info = (StdVideoEncodeH264PictureInfo) { + .flags = (StdVideoEncodeH264PictureInfoFlags) { + .IdrPicFlag = pic->type == FF_HW_PICTURE_TYPE_IDR, + .is_reference = pic->is_reference, + .no_output_of_prior_pics_flag = 0, + .long_term_reference_flag = 0, + .adaptive_ref_pic_marking_mode_flag = 0, // Filled in during setup_refs() + /* Reserved */ + }, + .seq_parameter_set_id = 0, + .pic_parameter_set_id = 0, + .idr_pic_id = hp->idr_pic_id, + .primary_pic_type = pic->type == FF_HW_PICTURE_TYPE_P ? STD_VIDEO_H264_PICTURE_TYPE_P : + pic->type == FF_HW_PICTURE_TYPE_B ? STD_VIDEO_H264_PICTURE_TYPE_B : + pic->type == FF_HW_PICTURE_TYPE_I ? STD_VIDEO_H264_PICTURE_TYPE_I : + STD_VIDEO_H264_PICTURE_TYPE_IDR, + .frame_num = hp->frame_num, + .PicOrderCnt = hp->pic_order_cnt, + .temporal_id = 0, /* ? */ + /* Reserved */ + .pRefLists = NULL, // Filled in during setup_refs + }; + encode_info->pNext = &hp->vkh264pic_info; + + hp->h264dpb_info = (StdVideoEncodeH264ReferenceInfo) { + .flags = (StdVideoEncodeH264ReferenceInfoFlags) { + .used_for_long_term_reference = 0, + /* Reserved */ + }, + .primary_pic_type = hp->h264pic_info.primary_pic_type, + .FrameNum = hp->h264pic_info.frame_num, + .PicOrderCnt = hp->h264pic_info.PicOrderCnt, + .long_term_pic_num = 0, + .long_term_frame_idx = 0, + .temporal_id = hp->h264pic_info.temporal_id, + }; + hp->vkh264dpb_info = (VkVideoEncodeH264DpbSlotInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_DPB_SLOT_INFO_KHR, + .pStdReferenceInfo = &hp->h264dpb_info, + }; + + vp->dpb_slot.pNext = &hp->vkh264dpb_info; + + ref_slot = (VkVideoReferenceSlotInfoKHR *)encode_info->pSetupReferenceSlot; + ref_slot->pNext = &hp->vkh264dpb_info; + + setup_refs(avctx, pic, encode_info); + + setup_slices(avctx, pic); + + return 0; +} + +static int init_profile(AVCodecContext *avctx, + VkVideoProfileInfoKHR *profile, void *pnext) +{ + VkResult ret; + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + FFVulkanContext *s = &ctx->s; + FFVulkanFunctions *vk = &ctx->s.vkfn; + VkVideoEncodeH264CapabilitiesKHR h264_caps = { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_CAPABILITIES_KHR, + }; + VkVideoEncodeCapabilitiesKHR enc_caps = { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_CAPABILITIES_KHR, + .pNext = &h264_caps, + }; + VkVideoCapabilitiesKHR caps = { + .sType = VK_STRUCTURE_TYPE_VIDEO_CAPABILITIES_KHR, + .pNext = &enc_caps, + }; + + /* In order of preference */ + int last_supported = AV_PROFILE_UNKNOWN; + static const int known_profiles[] = { + AV_PROFILE_H264_CONSTRAINED_BASELINE, + AV_PROFILE_H264_MAIN, + AV_PROFILE_H264_HIGH, + AV_PROFILE_H264_HIGH_10, + }; + + enc->profile = (VkVideoEncodeH264ProfileInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_PROFILE_INFO_KHR, + .pNext = pnext, + .stdProfileIdc = ff_vk_h264_profile_to_vk(avctx->profile), + }; + profile->pNext = &enc->profile; + + /* User has explicitly specified a profile. */ + if (avctx->profile != AV_PROFILE_UNKNOWN) + return 0; + + av_log(avctx, AV_LOG_DEBUG, "Supported profiles:\n"); + for (int i = 0; i < FF_ARRAY_ELEMS(known_profiles); i++) { + enc->profile.stdProfileIdc = ff_vk_h264_profile_to_vk(known_profiles[i]); + ret = vk->GetPhysicalDeviceVideoCapabilitiesKHR(s->hwctx->phys_dev, + profile, + &caps); + if (ret == VK_SUCCESS) { + av_log(avctx, AV_LOG_DEBUG, " %s\n", + avcodec_profile_name(avctx->codec_id, known_profiles[i])); + last_supported = known_profiles[i]; + } + } + + if (last_supported == AV_PROFILE_UNKNOWN) { + av_log(avctx, AV_LOG_ERROR, "No supported profiles for given format\n"); + return AVERROR(ENOTSUP); + } + + enc->profile.stdProfileIdc = ff_vk_h264_profile_to_vk(last_supported); + av_log(avctx, AV_LOG_VERBOSE, "Using profile %s\n", + avcodec_profile_name(avctx->codec_id, last_supported)); + avctx->profile = last_supported; + + return 0; +} + +static int init_enc_options(AVCodecContext *avctx) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + FFHWBaseEncodeH264Opts *unit_opts = &enc->unit_opts; + + if (avctx->rc_buffer_size) + unit_opts->hrd_buffer_size = avctx->rc_buffer_size; + else if (avctx->rc_max_rate > 0) + unit_opts->hrd_buffer_size = avctx->rc_max_rate; + else + unit_opts->hrd_buffer_size = avctx->bit_rate; + + if (avctx->rc_initial_buffer_occupancy) { + if (avctx->rc_initial_buffer_occupancy > unit_opts->hrd_buffer_size) { + av_log(avctx, AV_LOG_ERROR, "Invalid RC buffer settings: " + "must have initial buffer size (%d) <= " + "buffer size (%"PRId64").\n", + avctx->rc_initial_buffer_occupancy, unit_opts->hrd_buffer_size); + return AVERROR(EINVAL); + } + unit_opts->initial_buffer_fullness = avctx->rc_initial_buffer_occupancy; + } else { + unit_opts->initial_buffer_fullness = unit_opts->hrd_buffer_size * 3 / 4; + } + + if (enc->common.opts.rc_mode == VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR) { + enc->fixed_qp_p = av_clip(enc->common.opts.qp, 0, 51); + if (avctx->i_quant_factor > 0.0) + unit_opts->fixed_qp_idr = av_clip((avctx->i_quant_factor * enc->fixed_qp_p + + avctx->i_quant_offset) + 0.5, 0, 51); + else + unit_opts->fixed_qp_idr = enc->fixed_qp_p; + + if (avctx->b_quant_factor > 0.0) + enc->fixed_qp_b = av_clip((avctx->b_quant_factor * enc->fixed_qp_p + + avctx->b_quant_offset) + 0.5, 0, 51); + else + enc->fixed_qp_b = enc->fixed_qp_p; + + av_log(avctx, AV_LOG_DEBUG, "Using fixed QP = " + "%d / %d / %d for IDR- / P- / B-frames.\n", + unit_opts->fixed_qp_idr, enc->fixed_qp_p, enc->fixed_qp_b); + } else { + unit_opts->fixed_qp_idr = 26; + enc->fixed_qp_p = 26; + enc->fixed_qp_b = 26; + } + + return 0; +} + +static av_cold int init_sequence_headers(AVCodecContext *avctx) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + + FFHWBaseEncodeH264 *units = &enc->units; + FFHWBaseEncodeH264Opts *unit_opts = &enc->unit_opts; + + unit_opts->bit_rate = avctx->bit_rate; + unit_opts->mb_width = FFALIGN(avctx->width, 16) / 16; + unit_opts->mb_height = FFALIGN(avctx->height, 16) / 16; + unit_opts->flags = enc->unit_elems & UNIT_SEI_TIMING ? FF_HW_H264_SEI_TIMING : 0; + + /* cabac already set via an option */ + /* fixed_qp_idr initialized in init_enc_options() */ + /* hrd_buffer_size initialized in init_enc_options() */ + /* initial_buffer_fullness initialized in init_enc_options() */ + + err = ff_hw_base_encode_init_params_h264(&enc->common.base, avctx, + units, unit_opts); + if (err < 0) + return err; + + units->raw_sps.seq_scaling_matrix_present_flag = + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_SCALING_MATRIX_PRESENT_FLAG_SET_BIT_KHR); + units->raw_pps.pic_scaling_matrix_present_flag = + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_SCALING_MATRIX_PRESENT_FLAG_SET_BIT_KHR); + units->raw_pps.transform_8x8_mode_flag = + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_TRANSFORM_8X8_MODE_FLAG_SET_BIT_KHR); + + return 0; +} + +typedef struct VulkanH264Units { + StdVideoH264SequenceParameterSet vksps; + StdVideoH264ScalingLists vksps_scaling; + StdVideoH264HrdParameters vksps_vui_header; + StdVideoH264SequenceParameterSetVui vksps_vui; + + StdVideoH264PictureParameterSet vkpps; + StdVideoH264ScalingLists vkpps_scaling; +} VulkanH264Units; + +static av_cold int base_unit_to_vk(AVCodecContext *avctx, + VulkanH264Units *vk_units) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + FFHWBaseEncodeContext *base_ctx = &ctx->base; + + FFHWBaseEncodeH264 *units = &enc->units; + + H264RawSPS *sps = &units->raw_sps; + H264RawHRD *hrd = &sps->vui.nal_hrd_parameters; + StdVideoH264ScalingLists *vksps_scaling = &vk_units->vksps_scaling; + StdVideoH264HrdParameters *vksps_vui_header = &vk_units->vksps_vui_header; + StdVideoH264SequenceParameterSetVui *vksps_vui = &vk_units->vksps_vui; + StdVideoH264SequenceParameterSet *vksps = &vk_units->vksps; + + H264RawPPS *pps = &units->raw_pps; + StdVideoH264ScalingLists *vkpps_scaling = &vk_units->vkpps_scaling; + StdVideoH264PictureParameterSet *vkpps = &vk_units->vkpps; + + *vksps_scaling = (StdVideoH264ScalingLists) { + .scaling_list_present_mask = 0x0, // mask + .use_default_scaling_matrix_mask = 1, + }; + + *vksps_vui_header = (StdVideoH264HrdParameters) { + .cpb_cnt_minus1 = hrd->cpb_cnt_minus1, + .bit_rate_scale = hrd->bit_rate_scale, + .cpb_size_scale = hrd->cpb_size_scale, + /* Reserved */ + /* bit_rate/cpb_size/cbr_flag set below */ + .initial_cpb_removal_delay_length_minus1 = hrd->initial_cpb_removal_delay_length_minus1, + .cpb_removal_delay_length_minus1 = hrd->cpb_removal_delay_length_minus1, + .dpb_output_delay_length_minus1 = hrd->dpb_output_delay_length_minus1, + .time_offset_length = hrd->time_offset_length, + }; + + for (int i = 0; i < H264_MAX_CPB_CNT; i++) { + vksps_vui_header->bit_rate_value_minus1[i] = hrd->bit_rate_value_minus1[i]; + vksps_vui_header->cpb_size_value_minus1[i] = hrd->cpb_size_value_minus1[i]; + vksps_vui_header->cbr_flag[i] = hrd->cbr_flag[i]; + } + + *vksps_vui = (StdVideoH264SequenceParameterSetVui) { + .flags = (StdVideoH264SpsVuiFlags) { + .aspect_ratio_info_present_flag = sps->vui.aspect_ratio_info_present_flag, + .overscan_info_present_flag = sps->vui.overscan_info_present_flag, + .overscan_appropriate_flag = sps->vui.overscan_appropriate_flag, + .video_signal_type_present_flag = sps->vui.video_signal_type_present_flag, + .video_full_range_flag = sps->vui.video_full_range_flag, + .color_description_present_flag = sps->vui.colour_description_present_flag, + .chroma_loc_info_present_flag = sps->vui.chroma_loc_info_present_flag, + .timing_info_present_flag = sps->vui.timing_info_present_flag, + .fixed_frame_rate_flag = sps->vui.fixed_frame_rate_flag, + .bitstream_restriction_flag = sps->vui.bitstream_restriction_flag, + .nal_hrd_parameters_present_flag = sps->vui.nal_hrd_parameters_present_flag, + .vcl_hrd_parameters_present_flag = sps->vui.vcl_hrd_parameters_present_flag, + }, + .aspect_ratio_idc = sps->vui.aspect_ratio_idc, + .sar_width = sps->vui.sar_width, + .sar_height = sps->vui.sar_height, + .video_format = sps->vui.video_format, + .colour_primaries = sps->vui.colour_primaries, + .transfer_characteristics = sps->vui.transfer_characteristics, + .matrix_coefficients = sps->vui.matrix_coefficients, + .num_units_in_tick = sps->vui.num_units_in_tick, + .time_scale = sps->vui.time_scale, + .max_num_reorder_frames = sps->vui.max_num_reorder_frames, + .max_dec_frame_buffering = sps->vui.max_dec_frame_buffering, + .chroma_sample_loc_type_top_field = sps->vui.chroma_sample_loc_type_top_field, + .chroma_sample_loc_type_bottom_field = sps->vui.chroma_sample_loc_type_bottom_field, + /* Reserved */ + .pHrdParameters = vksps_vui_header, + }; + + *vksps = (StdVideoH264SequenceParameterSet) { + .flags = (StdVideoH264SpsFlags) { + .constraint_set0_flag = sps->constraint_set0_flag, + .constraint_set1_flag = sps->constraint_set1_flag, + .constraint_set2_flag = sps->constraint_set2_flag, + .constraint_set3_flag = sps->constraint_set3_flag, + .constraint_set4_flag = sps->constraint_set4_flag, + .constraint_set5_flag = sps->constraint_set5_flag, + .direct_8x8_inference_flag = sps->direct_8x8_inference_flag, + .mb_adaptive_frame_field_flag = sps->mb_adaptive_frame_field_flag, + .frame_mbs_only_flag = sps->frame_mbs_only_flag, + .delta_pic_order_always_zero_flag = sps->delta_pic_order_always_zero_flag, + .separate_colour_plane_flag = sps->separate_colour_plane_flag, + .gaps_in_frame_num_value_allowed_flag = sps->gaps_in_frame_num_allowed_flag, + .qpprime_y_zero_transform_bypass_flag = sps->qpprime_y_zero_transform_bypass_flag, + .frame_cropping_flag = sps->frame_cropping_flag, + .seq_scaling_matrix_present_flag = sps->seq_scaling_matrix_present_flag, + .vui_parameters_present_flag = sps->vui_parameters_present_flag, + }, + .profile_idc = ff_vk_h264_profile_to_vk(sps->profile_idc), + .level_idc = ff_vk_h264_level_to_vk(sps->level_idc), + .chroma_format_idc = sps->chroma_format_idc, + .seq_parameter_set_id = sps->seq_parameter_set_id, + .bit_depth_luma_minus8 = sps->bit_depth_luma_minus8, + .bit_depth_chroma_minus8 = sps->bit_depth_chroma_minus8, + .log2_max_frame_num_minus4 = sps->log2_max_frame_num_minus4, + .pic_order_cnt_type = sps->pic_order_cnt_type, + .offset_for_non_ref_pic = sps->offset_for_non_ref_pic, + .offset_for_top_to_bottom_field = sps->offset_for_top_to_bottom_field, + .log2_max_pic_order_cnt_lsb_minus4 = sps->log2_max_pic_order_cnt_lsb_minus4, + .num_ref_frames_in_pic_order_cnt_cycle = sps->num_ref_frames_in_pic_order_cnt_cycle, + .max_num_ref_frames = sps->max_num_ref_frames, + /* Reserved */ + .pic_width_in_mbs_minus1 = sps->pic_width_in_mbs_minus1, + .pic_height_in_map_units_minus1 = sps->pic_height_in_map_units_minus1, + .frame_crop_left_offset = sps->frame_crop_left_offset, + .frame_crop_right_offset = sps->frame_crop_right_offset, + .frame_crop_top_offset = sps->frame_crop_top_offset, + .frame_crop_bottom_offset = sps->frame_crop_bottom_offset, + /* Reserved */ + .pOffsetForRefFrame = sps->offset_for_ref_frame, + .pScalingLists = vksps_scaling, + .pSequenceParameterSetVui = vksps_vui, + }; + + *vkpps_scaling = (StdVideoH264ScalingLists) { + .scaling_list_present_mask = 0x0, // mask + .use_default_scaling_matrix_mask = 1, + }; + + *vkpps = (StdVideoH264PictureParameterSet) { + .flags = (StdVideoH264PpsFlags) { + .transform_8x8_mode_flag = pps->transform_8x8_mode_flag, + .redundant_pic_cnt_present_flag = pps->redundant_pic_cnt_present_flag, + .constrained_intra_pred_flag = pps->constrained_intra_pred_flag, + .deblocking_filter_control_present_flag = pps->deblocking_filter_control_present_flag, + .weighted_pred_flag = pps->weighted_pred_flag, + .bottom_field_pic_order_in_frame_present_flag = pps->bottom_field_pic_order_in_frame_present_flag, + .entropy_coding_mode_flag = pps->entropy_coding_mode_flag, + .pic_scaling_matrix_present_flag = pps->pic_scaling_matrix_present_flag, + }, + .seq_parameter_set_id = pps->seq_parameter_set_id, + .pic_parameter_set_id = pps->pic_parameter_set_id, + .num_ref_idx_l0_default_active_minus1 = pps->num_ref_idx_l0_default_active_minus1, + .num_ref_idx_l1_default_active_minus1 = pps->num_ref_idx_l1_default_active_minus1, + .weighted_bipred_idc = pps->weighted_bipred_idc, + .pic_init_qp_minus26 = pps->pic_init_qp_minus26, + .pic_init_qs_minus26 = pps->pic_init_qs_minus26, + .chroma_qp_index_offset = pps->chroma_qp_index_offset, + .second_chroma_qp_index_offset = pps->second_chroma_qp_index_offset, + .pScalingLists = vkpps_scaling, + }; + + return 0; +} + +static int create_session_params(AVCodecContext *avctx) +{ + int err; + VkResult ret; + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + FFVulkanContext *s = &ctx->s; + FFVulkanFunctions *vk = &ctx->s.vkfn; + + VulkanH264Units vk_units = { 0 }; + + VkVideoEncodeH264SessionParametersAddInfoKHR h264_params_info; + VkVideoEncodeH264SessionParametersCreateInfoKHR h264_params; + VkVideoSessionParametersCreateInfoKHR session_params_create; + + /* Convert it to Vulkan */ + err = base_unit_to_vk(avctx, &vk_units); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Unable to convert SPS/PPS units to Vulkan: %s\n", + av_err2str(err)); + return err; + } + + /* Destroy the session params */ + if (ctx->session_params) + vk->DestroyVideoSessionParametersKHR(s->hwctx->act_dev, + ctx->session_params, + s->hwctx->alloc); + + h264_params_info = (VkVideoEncodeH264SessionParametersAddInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_SESSION_PARAMETERS_ADD_INFO_KHR, + .pStdSPSs = &vk_units.vksps, + .stdSPSCount = 1, + .pStdPPSs = &vk_units.vkpps, + .stdPPSCount = 1, + }; + h264_params = (VkVideoEncodeH264SessionParametersCreateInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_SESSION_PARAMETERS_CREATE_INFO_KHR, + .maxStdSPSCount = 1, + .maxStdPPSCount = 1, + .pParametersAddInfo = &h264_params_info, + }; + session_params_create = (VkVideoSessionParametersCreateInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_SESSION_PARAMETERS_CREATE_INFO_KHR, + .pNext = &h264_params, + .videoSession = ctx->common.session, + .videoSessionParametersTemplate = NULL, + }; + + /* Create session parameters */ + ret = vk->CreateVideoSessionParametersKHR(s->hwctx->act_dev, &session_params_create, + s->hwctx->alloc, &ctx->session_params); + if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Unable to create Vulkan video session parameters: %s!\n", + ff_vk_ret2str(ret)); + return AVERROR_EXTERNAL; + } + + return 0; +} + +static int parse_feedback_units(AVCodecContext *avctx, + const uint8_t *data, size_t size, + int sps_override, int pps_override) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + + CodedBitstreamContext *cbs; + CodedBitstreamFragment au = { 0 }; + + err = ff_cbs_init(&cbs, AV_CODEC_ID_H264, avctx); + if (err < 0) + return err; + + err = ff_cbs_read(cbs, &au, data, size); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Unable to parse feedback units, bad drivers: %s\n", + av_err2str(err)); + return err; + } + + /* If PPS has an override, just copy it entirely. */ + if (pps_override) { + for (int i = 0; i < au.nb_units; i++) { + if (au.units[i].type == H264_NAL_PPS) { + H264RawPPS *pps = au.units[i].content; + memcpy(&enc->units.raw_pps, pps, sizeof(*pps)); + break; + } + } + } + + ff_cbs_fragment_free(&au); + ff_cbs_close(&cbs); + + return 0; +} + +static int init_base_units(AVCodecContext *avctx) +{ + int err; + VkResult ret; + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + FFVulkanContext *s = &ctx->s; + FFVulkanFunctions *vk = &ctx->s.vkfn; + + VkVideoEncodeH264SessionParametersGetInfoKHR h264_params_info; + VkVideoEncodeSessionParametersGetInfoKHR params_info; + VkVideoEncodeH264SessionParametersFeedbackInfoKHR h264_params_feedback; + VkVideoEncodeSessionParametersFeedbackInfoKHR params_feedback; + + void *data = NULL; + size_t data_size = 0; + + /* Generate SPS/PPS unit info */ + err = init_sequence_headers(avctx); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Unable to initialize SPS/PPS units: %s\n", + av_err2str(err)); + return err; + } + + /* Create session parameters from them */ + err = create_session_params(avctx); + if (err < 0) + return err; + + h264_params_info = (VkVideoEncodeH264SessionParametersGetInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_SESSION_PARAMETERS_GET_INFO_KHR, + .writeStdSPS = 1, + .writeStdPPS = 1, + .stdSPSId = enc->units.raw_sps.seq_parameter_set_id, + .stdPPSId = enc->units.raw_pps.pic_parameter_set_id, + }; + params_info = (VkVideoEncodeSessionParametersGetInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_SESSION_PARAMETERS_GET_INFO_KHR, + .pNext = &h264_params_info, + .videoSessionParameters = ctx->session_params, + }; + + h264_params_feedback = (VkVideoEncodeH264SessionParametersFeedbackInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_SESSION_PARAMETERS_FEEDBACK_INFO_KHR, + }; + params_feedback = (VkVideoEncodeSessionParametersFeedbackInfoKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_SESSION_PARAMETERS_FEEDBACK_INFO_KHR, + .pNext = &h264_params_feedback, + }; + + ret = vk->GetEncodedVideoSessionParametersKHR(s->hwctx->act_dev, ¶ms_info, + ¶ms_feedback, + &data_size, data); + if (ret == VK_INCOMPLETE || + (ret == VK_SUCCESS) && (data_size > 0)) { + data = av_mallocz(data_size); + if (!data) + return AVERROR(ENOMEM); + } else { + av_log(avctx, AV_LOG_ERROR, "Unable to get feedback for H.264 units = %lu\n", data_size); + return err; + } + + ret = vk->GetEncodedVideoSessionParametersKHR(s->hwctx->act_dev, ¶ms_info, + ¶ms_feedback, + &data_size, data); + if (ret != VK_SUCCESS) { + av_log(avctx, AV_LOG_ERROR, "Error writing feedback units\n"); + return err; + } + + av_log(avctx, AV_LOG_VERBOSE, "Feedback units written, overrides: %i (SPS: %i PPS: %i)\n", + params_feedback.hasOverrides, + h264_params_feedback.hasStdSPSOverrides, + h264_params_feedback.hasStdPPSOverrides); + + params_feedback.hasOverrides = 1; + h264_params_feedback.hasStdPPSOverrides = 1; + + /* No need to sync any overrides */ + if (!params_feedback.hasOverrides) + return 0; + + /* Parse back tne units and override */ + err = parse_feedback_units(avctx, data, data_size, + h264_params_feedback.hasStdSPSOverrides, + h264_params_feedback.hasStdPPSOverrides); + if (err < 0) + return err; + + /* Create final session parameters */ + err = create_session_params(avctx); + if (err < 0) + return err; + + return 0; +} + +static int vulkan_encode_h264_add_nal(AVCodecContext *avctx, + CodedBitstreamFragment *au, + void *nal_unit) +{ + H264RawNALUnitHeader *header = nal_unit; + + int err = ff_cbs_insert_unit_content(au, -1, + header->nal_unit_type, nal_unit, NULL); + if (err < 0) + av_log(avctx, AV_LOG_ERROR, "Failed to add NAL unit: " + "type = %d.\n", header->nal_unit_type); + + return err; +} + +static int write_access_unit(AVCodecContext *avctx, + uint8_t *data, size_t *data_len, + CodedBitstreamFragment *au) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + + int err = ff_cbs_write_fragment_data(enc->cbs, au); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to write packed header.\n"); + return err; + } + + if (*data_len < au->data_size) { + av_log(avctx, AV_LOG_ERROR, "Access unit too large: %zu < %zu.\n", + *data_len, au->data_size); + return AVERROR(ENOSPC); + } + + memcpy(data, au->data, au->data_size); + *data_len = au->data_size; + + return 0; +} + +static int write_sequence_header(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic, + uint8_t *data, size_t *data_len) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + VulkanEncodeH264Picture *hp = base_pic ? base_pic->codec_priv : NULL; + CodedBitstreamFragment *au = &enc->current_access_unit; + + if (hp && hp->units_needed & UNIT_AUD) { + err = vulkan_encode_h264_add_nal(avctx, au, &enc->raw_aud); + if (err < 0) + goto fail; + hp->units_needed &= ~UNIT_AUD; + } + + err = vulkan_encode_h264_add_nal(avctx, au, &enc->units.raw_sps); + if (err < 0) + goto fail; + + err = vulkan_encode_h264_add_nal(avctx, au, &enc->units.raw_pps); + if (err < 0) + goto fail; + + err = write_access_unit(avctx, data, data_len, au); +fail: + ff_cbs_fragment_reset(au); + return err; +} + +static int write_extra_headers(AVCodecContext *avctx, + FFHWBaseEncodePicture *base_pic, + uint8_t *data, size_t *data_len) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + VulkanEncodeH264Picture *hp = base_pic->codec_priv; + CodedBitstreamFragment *au = &enc->current_access_unit; + + if (hp->units_needed & UNIT_AUD) { + err = vulkan_encode_h264_add_nal(avctx, au, &enc->raw_aud); + if (err < 0) + goto fail; + hp->units_needed &= ~UNIT_AUD; + } + + if (hp->units_needed & UNIT_SEI_IDENTIFIER) { + err = ff_cbs_sei_add_message(enc->cbs, au, 1, + SEI_TYPE_USER_DATA_UNREGISTERED, + &enc->sei_identifier, NULL); + if (err < 0) + goto fail; + } + + if (hp->units_needed & UNIT_SEI_TIMING) { + if (base_pic->type == FF_HW_PICTURE_TYPE_IDR) { + err = ff_cbs_sei_add_message(enc->cbs, au, 1, + SEI_TYPE_BUFFERING_PERIOD, + &enc->units.sei_buffering_period, NULL); + if (err < 0) + goto fail; + } + err = ff_cbs_sei_add_message(enc->cbs, au, 1, + SEI_TYPE_PIC_TIMING, + &enc->sei_pic_timing, NULL); + if (err < 0) + goto fail; + } + + if (hp->units_needed & UNIT_SEI_RECOVERY) { + err = ff_cbs_sei_add_message(enc->cbs, au, 1, + SEI_TYPE_RECOVERY_POINT, + &enc->sei_recovery_point, NULL); + if (err < 0) + goto fail; + } + + if (hp->units_needed & UNIT_SEI_A53_CC) { + err = ff_cbs_sei_add_message(enc->cbs, au, 1, + SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35, + &enc->sei_a53cc, NULL); + if (err < 0) + goto fail; + } + + if (hp->units_needed) { + err = write_access_unit(avctx, data, data_len, au); + if (err < 0) + goto fail; + } + +fail: + ff_cbs_fragment_reset(au); + return err; +} + +static int write_filler(AVCodecContext *avctx, uint32_t filler, + uint8_t *data, size_t *data_len) +{ + int err; + VulkanEncodeH264Context *enc = avctx->priv_data; + CodedBitstreamFragment *au = &enc->current_access_unit; + + H264RawFiller raw_filler = { + .nal_unit_header = { + .nal_unit_type = H264_NAL_FILLER_DATA, + }, + .filler_size = filler, + }; + + err = vulkan_encode_h264_add_nal(avctx, au, &raw_filler); + if (err < 0) + goto fail; + + err = write_access_unit(avctx, data, data_len, au); +fail: + ff_cbs_fragment_reset(au); + return err; +} + +static const FFVulkanCodec enc_cb = { + .flags = FF_HW_FLAG_B_PICTURES | + FF_HW_FLAG_B_PICTURE_REFERENCES | + FF_HW_FLAG_NON_IDR_KEY_PICTURES | + FF_HW_FLAG_SLICE_CONTROL, + .picture_priv_data_size = sizeof(VulkanEncodeH264Picture), + .filler_header_size = 6, + .init_profile = init_profile, + .init_pic_rc = init_pic_rc, + .init_pic_params = init_pic_params, + .write_sequence_headers = write_sequence_header, + .write_extra_headers = write_extra_headers, + .write_filler = write_filler, +}; + +static av_cold int vulkan_encode_h264_init(AVCodecContext *avctx) +{ + int err, ref_l0, ref_l1; + VulkanEncodeH264Context *enc = avctx->priv_data; + FFVulkanEncodeContext *ctx = &enc->common; + FFVulkanContext *s = &ctx->s; + FFHWBaseEncodeContext *base_ctx = &ctx->base; + int flags; + + if (avctx->profile == AV_PROFILE_UNKNOWN) + avctx->profile = enc->common.opts.profile; + + enc->caps = (VkVideoEncodeH264CapabilitiesKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_CAPABILITIES_KHR, + }; + + enc->quality_props = (VkVideoEncodeH264QualityLevelPropertiesKHR) { + .sType = VK_STRUCTURE_TYPE_VIDEO_ENCODE_H264_QUALITY_LEVEL_PROPERTIES_KHR, + }; + + err = ff_vulkan_encode_init(avctx, &enc->common, + &ff_vk_enc_h264_desc, &enc_cb, + &enc->caps, &enc->quality_props); + if (err < 0) + return err; + + av_log(avctx, AV_LOG_VERBOSE, "H264 encoder capabilities:\n"); + av_log(avctx, AV_LOG_VERBOSE, " Standard capability flags:\n"); + av_log(avctx, AV_LOG_VERBOSE, " separate_color_plane: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_SEPARATE_COLOR_PLANE_FLAG_SET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " qprime_y_zero_transform_bypass: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_QPPRIME_Y_ZERO_TRANSFORM_BYPASS_FLAG_SET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " scaling_lists: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_SCALING_MATRIX_PRESENT_FLAG_SET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " chroma_qp_index_offset: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_CHROMA_QP_INDEX_OFFSET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " second_chroma_qp_index_offset: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_SECOND_CHROMA_QP_INDEX_OFFSET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " pic_init_qp: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_PIC_INIT_QP_MINUS26_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " weighted:%s%s%s\n", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_WEIGHTED_PRED_FLAG_SET_BIT_KHR ? + " pred" : "", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_EXPLICIT_BIT_KHR ? + " bipred_explicit" : "", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_WEIGHTED_BIPRED_IDC_IMPLICIT_BIT_KHR ? + " bipred_implicit" : ""); + av_log(avctx, AV_LOG_VERBOSE, " 8x8_transforms: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_TRANSFORM_8X8_MODE_FLAG_SET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " disable_direct_spatial_mv_pred: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_DIRECT_SPATIAL_MV_PRED_FLAG_UNSET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " coder:%s%s\n", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_UNSET_BIT_KHR ? + " cabac" : "", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_ENTROPY_CODING_MODE_FLAG_SET_BIT_KHR ? + " cavlc" : ""); + av_log(avctx, AV_LOG_VERBOSE, " direct_8x8_inference: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_DIRECT_8X8_INFERENCE_FLAG_UNSET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " constrained_intra_pred: %i\n", + !!(enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_CONSTRAINED_INTRA_PRED_FLAG_SET_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " deblock:%s%s%s\n", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_DISABLED_BIT_KHR ? + " filter_disabling" : "", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_ENABLED_BIT_KHR ? + " filter_enabling" : "", + enc->caps.stdSyntaxFlags & VK_VIDEO_ENCODE_H264_STD_DEBLOCKING_FILTER_PARTIAL_BIT_KHR ? + " filter_partial" : ""); + + av_log(avctx, AV_LOG_VERBOSE, " Capability flags:\n"); + av_log(avctx, AV_LOG_VERBOSE, " hdr_compliance: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_HRD_COMPLIANCE_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " pred_weight_table_generated: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_PREDICTION_WEIGHT_TABLE_GENERATED_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " row_unaligned_slice: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_ROW_UNALIGNED_SLICE_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " different_slice_type: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_DIFFERENT_SLICE_TYPE_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " b_frame_in_l0_list: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L0_LIST_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " b_frame_in_l1_list: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_B_FRAME_IN_L1_LIST_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " per_pict_type_min_max_qp: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_PER_PICTURE_TYPE_MIN_MAX_QP_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " per_slice_constant_qp: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_PER_SLICE_CONSTANT_QP_BIT_KHR)); + av_log(avctx, AV_LOG_VERBOSE, " generate_prefix_nalu: %i\n", + !!(enc->caps.flags & VK_VIDEO_ENCODE_H264_CAPABILITY_GENERATE_PREFIX_NALU_BIT_KHR)); + + av_log(avctx, AV_LOG_VERBOSE, " Capabilities:\n"); + av_log(avctx, AV_LOG_VERBOSE, " maxLevelIdc: %i\n", + enc->caps.maxLevelIdc); + av_log(avctx, AV_LOG_VERBOSE, " maxSliceCount: %i\n", + enc->caps.maxSliceCount); + av_log(avctx, AV_LOG_VERBOSE, " max(P/B)PictureL0ReferenceCount: %i P's; %i B's\n", + enc->caps.maxPPictureL0ReferenceCount, + enc->caps.maxBPictureL0ReferenceCount); + av_log(avctx, AV_LOG_VERBOSE, " maxL1ReferenceCount: %i\n", + enc->caps.maxL1ReferenceCount); + av_log(avctx, AV_LOG_VERBOSE, " maxTemporalLayerCount: %i\n", + enc->caps.maxTemporalLayerCount); + av_log(avctx, AV_LOG_VERBOSE, " expectDyadicTemporalLayerPattern: %i\n", + enc->caps.expectDyadicTemporalLayerPattern); + av_log(avctx, AV_LOG_VERBOSE, " min/max Qp: [%i, %i]\n", + enc->caps.maxQp, enc->caps.minQp); + av_log(avctx, AV_LOG_VERBOSE, " prefersGopRemainingFrames: %i\n", + enc->caps.prefersGopRemainingFrames); + av_log(avctx, AV_LOG_VERBOSE, " requiresGopRemainingFrames: %i\n", + enc->caps.requiresGopRemainingFrames); + + err = init_enc_options(avctx); + if (err < 0) + return err; + + flags = ctx->codec->flags; + if (!enc->caps.maxPPictureL0ReferenceCount && + !enc->caps.maxBPictureL0ReferenceCount && + !enc->caps.maxL1ReferenceCount) { + /* Intra-only */ + flags |= FF_HW_FLAG_INTRA_ONLY; + ref_l0 = ref_l1 = 0; + } else if (!enc->caps.maxPPictureL0ReferenceCount) { + /* No P-frames? How. */ + base_ctx->p_to_gpb = 1; + ref_l0 = enc->caps.maxBPictureL0ReferenceCount; + ref_l1 = enc->caps.maxL1ReferenceCount; + } else if (!enc->caps.maxBPictureL0ReferenceCount && + !enc->caps.maxL1ReferenceCount) { + /* No B-frames */ + flags &= ~(FF_HW_FLAG_B_PICTURES | FF_HW_FLAG_B_PICTURE_REFERENCES); + ref_l0 = enc->caps.maxPPictureL0ReferenceCount; + ref_l1 = 0; + } else { + /* P and B frames */ + ref_l0 = FFMIN(enc->caps.maxPPictureL0ReferenceCount, + enc->caps.maxBPictureL0ReferenceCount); + ref_l1 = enc->caps.maxL1ReferenceCount; + } + + err = ff_hw_base_init_gop_structure(base_ctx, avctx, ref_l0, ref_l1, + flags, 0); + if (err < 0) + return err; + + base_ctx->output_delay = base_ctx->b_per_p; + base_ctx->decode_delay = base_ctx->max_b_depth; + + /* Create units and session parameters */ + err = init_base_units(avctx); + if (err < 0) + return err; + + /* Prepare SEI */ + if (enc->unit_elems & UNIT_SEI_IDENTIFIER) { + int len; + + memcpy(enc->sei_identifier.uuid_iso_iec_11578, + vulkan_encode_h264_sei_identifier_uuid, + sizeof(enc->sei_identifier.uuid_iso_iec_11578)); + + len = snprintf(NULL, 0, + "%s / Vulkan video %i.%i.%i / %s %i.%i.%i / %s", + LIBAVCODEC_IDENT, + CODEC_VER(ff_vk_enc_h264_desc.ext_props.specVersion), + s->driver_props.driverName, + CODEC_VER(s->props.properties.driverVersion), + s->props.properties.deviceName); + + if (len >= 0) { + enc->sei_identifier_string = av_malloc(len + 1); + if (!enc->sei_identifier_string) + return AVERROR(ENOMEM); + + len = snprintf(enc->sei_identifier_string, len + 1, + "%s / Vulkan video %i.%i.%i / %s %i.%i.%i / %s", + LIBAVCODEC_IDENT, + CODEC_VER(ff_vk_enc_h264_desc.ext_props.specVersion), + s->driver_props.driverName, + CODEC_VER(s->props.properties.driverVersion), + s->props.properties.deviceName); + + enc->sei_identifier.data = enc->sei_identifier_string; + enc->sei_identifier.data_length = len + 1; + } + } + + /* Init CBS */ + err = ff_cbs_init(&enc->cbs, AV_CODEC_ID_H264, avctx); + if (err < 0) + return err; + + /* Write extradata if needed */ + if (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) { + uint8_t data[4096]; + size_t data_len = sizeof(data); + + err = write_sequence_header(avctx, NULL, data, &data_len); + if (err < 0) { + av_log(avctx, AV_LOG_ERROR, "Failed to write sequence header " + "for extradata: %d.\n", err); + return err; + } else { + avctx->extradata_size = data_len; + avctx->extradata = av_mallocz(avctx->extradata_size + + AV_INPUT_BUFFER_PADDING_SIZE); + if (!avctx->extradata) { + err = AVERROR(ENOMEM); + return err; + } + memcpy(avctx->extradata, data, avctx->extradata_size); + } + } + + return 0; +} + +static av_cold int vulkan_encode_h264_close(AVCodecContext *avctx) +{ + VulkanEncodeH264Context *enc = avctx->priv_data; + ff_vulkan_encode_uninit(&enc->common); + return 0; +} + +#define OFFSET(x) offsetof(VulkanEncodeH264Context, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_ENCODING_PARAM) +static const AVOption vulkan_encode_h264_options[] = { + HW_BASE_ENCODE_COMMON_OPTIONS, + VULKAN_ENCODE_COMMON_OPTIONS, + + { "profile", "Set profile (profile_idc and constraint_set*_flag)", + OFFSET(common.opts.profile), AV_OPT_TYPE_INT, + { .i64 = AV_PROFILE_UNKNOWN }, AV_PROFILE_UNKNOWN, 0xffff, FLAGS, .unit = "profile" }, + +#define PROFILE(name, value) name, NULL, 0, AV_OPT_TYPE_CONST, \ + { .i64 = value }, 0, 0, FLAGS, .unit = "profile" + { PROFILE("constrained_baseline", AV_PROFILE_H264_CONSTRAINED_BASELINE) }, + { PROFILE("main", AV_PROFILE_H264_MAIN) }, + { PROFILE("high", AV_PROFILE_H264_HIGH) }, + { PROFILE("high444p", AV_PROFILE_H264_HIGH_10) }, +#undef PROFILE + + { "level", "Set level (level_idc)", + OFFSET(common.opts.level), AV_OPT_TYPE_INT, + { .i64 = AV_LEVEL_UNKNOWN }, AV_LEVEL_UNKNOWN, 0xff, FLAGS, .unit = "level" }, + +#define LEVEL(name, value) name, NULL, 0, AV_OPT_TYPE_CONST, \ + { .i64 = value }, 0, 0, FLAGS, .unit = "level" + { LEVEL("1", 10) }, + { LEVEL("1.1", 11) }, + { LEVEL("1.2", 12) }, + { LEVEL("1.3", 13) }, + { LEVEL("2", 20) }, + { LEVEL("2.1", 21) }, + { LEVEL("2.2", 22) }, + { LEVEL("3", 30) }, + { LEVEL("3.1", 31) }, + { LEVEL("3.2", 32) }, + { LEVEL("4", 40) }, + { LEVEL("4.1", 41) }, + { LEVEL("4.2", 42) }, + { LEVEL("5", 50) }, + { LEVEL("5.1", 51) }, + { LEVEL("5.2", 52) }, + { LEVEL("6", 60) }, + { LEVEL("6.1", 61) }, + { LEVEL("6.2", 62) }, +#undef LEVEL + + { "coder", "Entropy coder type", OFFSET(unit_opts.cabac), AV_OPT_TYPE_INT, { .i64 = 1 }, 0, 1, FLAGS, "coder" }, + { "cabac", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 1 }, INT_MIN, INT_MAX, FLAGS, "coder" }, + { "vlc", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0 }, INT_MIN, INT_MAX, FLAGS, "coder" }, + + { "units", "Set units to include", OFFSET(unit_elems), AV_OPT_TYPE_FLAGS, { .i64 = UNIT_AUD | UNIT_SEI_IDENTIFIER | UNIT_SEI_RECOVERY | UNIT_SEI_TIMING | UNIT_SEI_A53_CC }, 0, INT_MAX, FLAGS, "units" }, + { "aud", "Include AUD units", 0, AV_OPT_TYPE_CONST, { .i64 = UNIT_AUD }, INT_MIN, INT_MAX, FLAGS, "units" }, + { "identifier", "Include encoder version identifier", 0, AV_OPT_TYPE_CONST, { .i64 = UNIT_SEI_IDENTIFIER }, INT_MIN, INT_MAX, FLAGS, "units" }, + { "timing", "Include timing parameters (buffering_period and pic_timing)", 0, AV_OPT_TYPE_CONST, { .i64 = UNIT_SEI_TIMING }, INT_MIN, INT_MAX, FLAGS, "units" }, + { "recovery", "Include recovery points where appropriate", 0, AV_OPT_TYPE_CONST, { .i64 = UNIT_SEI_RECOVERY }, INT_MIN, INT_MAX, FLAGS, "units" }, + { "a53_cc", "Include A/53 caption data", 0, AV_OPT_TYPE_CONST, { .i64 = UNIT_SEI_A53_CC }, INT_MIN, INT_MAX, FLAGS, "units" }, + + { NULL }, +}; + +static const FFCodecDefault vulkan_encode_h264_defaults[] = { + { "b", "0" }, + { "bf", "2" }, + { "g", "300" }, + { "i_qfactor", "1" }, + { "i_qoffset", "0" }, + { "b_qfactor", "1" }, + { "b_qoffset", "0" }, + { "qmin", "-1" }, + { "qmax", "-1" }, + { NULL }, +}; + +static const AVClass vulkan_encode_h264_class = { + .class_name = "h264_vulkan", + .item_name = av_default_item_name, + .option = vulkan_encode_h264_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const FFCodec ff_h264_vulkan_encoder = { + .p.name = "h264_vulkan", + CODEC_LONG_NAME("H.264/AVC (Vulkan)"), + .p.type = AVMEDIA_TYPE_VIDEO, + .p.id = AV_CODEC_ID_H264, + .priv_data_size = sizeof(VulkanEncodeH264Context), + .init = &vulkan_encode_h264_init, + FF_CODEC_RECEIVE_PACKET_CB(&ff_vulkan_encode_receive_packet), + .close = &vulkan_encode_h264_close, + .p.priv_class = &vulkan_encode_h264_class, + .p.capabilities = AV_CODEC_CAP_DELAY | + AV_CODEC_CAP_HARDWARE | + AV_CODEC_CAP_DR1 | + AV_CODEC_CAP_ENCODER_FLUSH | + AV_CODEC_CAP_ENCODER_REORDERED_OPAQUE, + .caps_internal = FF_CODEC_CAP_INIT_CLEANUP, + .defaults = vulkan_encode_h264_defaults, + .p.pix_fmts = (const enum AVPixelFormat[]) { + AV_PIX_FMT_VULKAN, + AV_PIX_FMT_NONE, + }, + .hw_configs = ff_vulkan_encode_hw_configs, + .p.wrapper_name = "vulkan", +};