From patchwork Sat Nov 13 21:09:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: rcombs X-Patchwork-Id: 31408 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp3678559iob; Sat, 13 Nov 2021 13:10:47 -0800 (PST) X-Google-Smtp-Source: ABdhPJwmitKv+z7BxCMOqJfpuE5sJb/Eq88nGxFAWMHid993LzRq+AQBrX3o4JhMrHxxnplH3gEn X-Received: by 2002:a17:906:5d0b:: with SMTP id g11mr31341436ejt.295.1636837847277; Sat, 13 Nov 2021 13:10:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636837847; cv=none; d=google.com; s=arc-20160816; b=Hc4/sCSYptVnnLf2Xg68QlEceWHGx3R7e/qDk8Ii//d2WiGQaFGSyH1UOVUpRIe78x j8hB7fvTALlFBucp6AypeYiTS/QwTnVt7XiI0mtVTm4cHp/z4HSVzmhOllRVX5ixwa5D EEeIqJiIGd/oR00GT0uUimnsouFCAl0Xhn2S20nAP7lXgyQDPGyZH8qI2ZCgIeTXxgsS dRYvxTW5skBCWr5icLuExkE97CYByPQHOpIMCXOYauEZL9sfDg/BJEPxNHbRpShzOf9W vXmgEKrY2LnqN+eCmJ7K0nV4LNVr9sxExgIhiE9qxWe/X2Bl7Sjrvoijach8LklmNVQW PtPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=LVEs8mEJgtiG2DgqzwLekAc+ZqY4e4grjJmiHs5u/U4=; b=SjAkHb/ZPDkO/sD2SQc4gTlfmEBvd6LBheEcVxfDWT/epMF21PfHKq6BpLrROdMRFm zqN0eoJD3DEtXCpyVexK8BbMrIYofDbBrVUm6Qn6bWwgTZ6ZdhZWZZpoCNpUPf1yYeS7 0Gymf19X95o7A2OZGsdjySpAKrWcVEoFTWQSDUmWISO0uiSC5dB3PNHFX1oSymeNBeOX 4wtPuXLXsUb6lMYCqR3kSTO1mvUBWc9f4l6420uA4w8+QwY6uezgi8gmGjMRNod83IOX irzo0H3asAFRUQlM6Qwk7EY6CcMtYlybpMKRJ8dAhmOOrfoo5o9pbPEQZCyUyrYatuLq +gRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=PmOXn4NF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id dp7si24293883ejc.755.2021.11.13.13.10.46; Sat, 13 Nov 2021 13:10:47 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=PmOXn4NF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 43FC268AE4B; Sat, 13 Nov 2021 23:09:37 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-il1-f176.google.com (mail-il1-f176.google.com [209.85.166.176]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id BA64168AE42 for ; Sat, 13 Nov 2021 23:09:32 +0200 (EET) Received: by mail-il1-f176.google.com with SMTP id k1so12431011ilo.7 for ; Sat, 13 Nov 2021 13:09:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rcombs.me; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=B4oMpUujpqWZNoBl3PvzPR8lWGuG8LoR44X5xwo6Rls=; b=PmOXn4NFDZPNugDxdq1ZJM9rvwDE7hc2GQ8UdV4gdqqNWggJZEf+DDyKnW90LlwvE8 trY+OhOkGxO3VUJ6VGQluRzzTJLN+nWeTRVq3PGh8rkSfIIRDqQarDudQ9KnJqESJM3w Vdz2sYJPBMkCtMN4oZ+59Oz+DkOEkc+uqQ2Xt/YurTlvQQAFBenSh6+0cuxbw4LHp5LV fAgbeOqWpgkza7uixzEPzy8PJkjCgzJ5KAu9xX83RJrdkf0yPIBBzG2+7zP32HjhGo2s yVhy4JgnP/4DGqdhZm1Gxarp+ZvVPZBE/N7CIz3UJ6NLTCIU+PkNBeHEDtbvPYaoUsK6 snkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B4oMpUujpqWZNoBl3PvzPR8lWGuG8LoR44X5xwo6Rls=; b=lZEbpB4AiEVxtAeHj2u4Xlk5V7ZPKtcjolOHwJk5dzy+ysxfkGtb7La+o4oXdLvnda kUa0pB0k99IDXei+0GSGVPBfpHsANz56DzCnYjUMSBt1M9tJMGicDQ2EOasviYaOn7Zh po9m9K949J/PvXyDurz8A+TJmyHtuc9sNm0bByCljqEC6yF7EtCR6gb5ohprFXx1sFUh wMEmnnlQHfEZlL0WVQ3SIpAa5coH+p156+VeckqpLUrtQ3RrDfkXDdG2fLE31zbzabOA 9lTzeVHFMHEivP44O2B3y1CiWxMQeXlvMC2dLI1GtnwowD7FvMd0dGAIH1Up+YTyeqP3 5KBw== X-Gm-Message-State: AOAM531di+RdRVxPKOuVBoWx3UlPtoQQLbSD6k/CVuDkvHZNWd0K4tb1 Jm5vLR5oEe4RbzN20WZGAD9fjaUVYqhsUKs= X-Received: by 2002:a05:6e02:20e4:: with SMTP id q4mr14508177ilv.71.1636837770749; Sat, 13 Nov 2021 13:09:30 -0800 (PST) Received: from rcombs-mbp.localdomain ([2601:243:2000:5ac:516:f322:7db9:ae0a]) by smtp.gmail.com with ESMTPSA id j23sm5243896iog.53.2021.11.13.13.09.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Nov 2021 13:09:30 -0800 (PST) From: rcombs To: ffmpeg-devel@ffmpeg.org Date: Sat, 13 Nov 2021 15:09:13 -0600 Message-Id: <20211113210916.49167-12-rcombs@rcombs.me> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211113210916.49167-1-rcombs@rcombs.me> References: <20211113210916.49167-1-rcombs@rcombs.me> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 12/15] lavc/videotoolbox: add VP9 hardware acceleration X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 4HFOvTqjzCme On M1 Max, this supports profiles 0 and 2, but not 1 and 3. --- configure | 4 + libavcodec/Makefile | 1 + libavcodec/hwaccels.h | 1 + libavcodec/videotoolbox.c | 12 +++ libavcodec/videotoolbox_vp9.c | 140 ++++++++++++++++++++++++++++++++++ libavcodec/vp9.c | 9 ++- libavcodec/vt_internal.h | 1 + 7 files changed, 167 insertions(+), 1 deletion(-) create mode 100644 libavcodec/videotoolbox_vp9.c diff --git a/configure b/configure index 97c0133f8c..2c6dd59e62 100755 --- a/configure +++ b/configure @@ -2326,6 +2326,7 @@ TOOLCHAIN_FEATURES=" TYPES_LIST=" kCMVideoCodecType_HEVC kCMVideoCodecType_HEVCWithAlpha + kCMVideoCodecType_VP9 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange @@ -3081,6 +3082,8 @@ vp9_vaapi_hwaccel_deps="vaapi VADecPictureParameterBufferVP9_bit_depth" vp9_vaapi_hwaccel_select="vp9_decoder" vp9_vdpau_hwaccel_deps="vdpau VdpPictureInfoVP9" vp9_vdpau_hwaccel_select="vp9_decoder" +vp9_videotoolbox_hwaccel_deps="videotoolbox" +vp9_videotoolbox_hwaccel_select="vp9_decoder" wmv3_d3d11va_hwaccel_select="vc1_d3d11va_hwaccel" wmv3_d3d11va2_hwaccel_select="vc1_d3d11va2_hwaccel" wmv3_dxva2_hwaccel_select="vc1_dxva2_hwaccel" @@ -6282,6 +6285,7 @@ enabled videotoolbox && { check_lib coreservices CoreServices/CoreServices.h UTGetOSTypeFromString "-framework CoreServices" check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_HEVC "-framework CoreMedia" check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_HEVCWithAlpha "-framework CoreMedia" + check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_VP9 "-framework CoreMedia" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange "-framework CoreVideo" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange "-framework CoreVideo" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange "-framework CoreVideo" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 14fbd2ecbc..32b35ebe75 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -975,6 +975,7 @@ OBJS-$(CONFIG_VP9_DXVA2_HWACCEL) += dxva2_vp9.o OBJS-$(CONFIG_VP9_NVDEC_HWACCEL) += nvdec_vp9.o OBJS-$(CONFIG_VP9_VAAPI_HWACCEL) += vaapi_vp9.o OBJS-$(CONFIG_VP9_VDPAU_HWACCEL) += vdpau_vp9.o +OBJS-$(CONFIG_VP9_VIDEOTOOLBOX_HWACCEL) += videotoolbox_vp9.o OBJS-$(CONFIG_VP8_QSV_HWACCEL) += qsvdec.o # libavformat dependencies diff --git a/libavcodec/hwaccels.h b/libavcodec/hwaccels.h index 8e54cf73f9..65e778f3e4 100644 --- a/libavcodec/hwaccels.h +++ b/libavcodec/hwaccels.h @@ -74,6 +74,7 @@ extern const AVHWAccel ff_vp9_dxva2_hwaccel; extern const AVHWAccel ff_vp9_nvdec_hwaccel; extern const AVHWAccel ff_vp9_vaapi_hwaccel; extern const AVHWAccel ff_vp9_vdpau_hwaccel; +extern const AVHWAccel ff_vp9_videotoolbox_hwaccel; extern const AVHWAccel ff_wmv3_d3d11va_hwaccel; extern const AVHWAccel ff_wmv3_d3d11va2_hwaccel; extern const AVHWAccel ff_wmv3_dxva2_hwaccel; diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 0666446dbd..2e9af2ab93 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -46,6 +46,10 @@ enum { kCMVideoCodecType_HEVC = 'hvc1' }; #endif +#if !HAVE_KCMVIDEOCODECTYPE_VP9 +enum { kCMVideoCodecType_VP9 = 'vp09' }; +#endif + #define VIDEOTOOLBOX_ESDS_EXTRADATA_PADDING 12 typedef struct VTHWFrame { @@ -814,6 +818,11 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec if (data) CFDictionarySetValue(avc_info, CFSTR("hvcC"), data); break; + case kCMVideoCodecType_VP9 : + data = ff_videotoolbox_vpcc_extradata_create(avctx); + if (data) + CFDictionarySetValue(avc_info, CFSTR("vpcC"), data); + break; default: break; } @@ -861,6 +870,9 @@ static int videotoolbox_start(AVCodecContext *avctx) case AV_CODEC_ID_MPEG4 : videotoolbox->cm_codec_type = kCMVideoCodecType_MPEG4Video; break; + case AV_CODEC_ID_VP9 : + videotoolbox->cm_codec_type = kCMVideoCodecType_VP9; + break; default : break; } diff --git a/libavcodec/videotoolbox_vp9.c b/libavcodec/videotoolbox_vp9.c new file mode 100644 index 0000000000..1b6d08f00b --- /dev/null +++ b/libavcodec/videotoolbox_vp9.c @@ -0,0 +1,140 @@ +/* + * Videotoolbox hardware acceleration for VP9 + * + * copyright (c) 2021 rcombs + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "config.h" +#include "videotoolbox.h" +#include "libavutil/hwcontext_videotoolbox.h" +#include "vt_internal.h" +#include "libavutil/avutil.h" +#include "libavutil/frame.h" +#include "libavutil/hwcontext.h" +#include "libavutil/intreadwrite.h" +#include "libavutil/pixdesc.h" +#include "decode.h" +#include "internal.h" +#include "vp9shared.h" + +enum VPX_CHROMA_SUBSAMPLING +{ + VPX_SUBSAMPLING_420_VERTICAL = 0, + VPX_SUBSAMPLING_420_COLLOCATED_WITH_LUMA = 1, + VPX_SUBSAMPLING_422 = 2, + VPX_SUBSAMPLING_444 = 3, +}; + +static int get_vpx_chroma_subsampling(enum AVPixelFormat pixel_format, + enum AVChromaLocation chroma_location) +{ + int chroma_w, chroma_h; + if (av_pix_fmt_get_chroma_sub_sample(pixel_format, &chroma_w, &chroma_h) == 0) { + if (chroma_w == 1 && chroma_h == 1) { + return (chroma_location == AVCHROMA_LOC_LEFT) + ? VPX_SUBSAMPLING_420_VERTICAL + : VPX_SUBSAMPLING_420_COLLOCATED_WITH_LUMA; + } else if (chroma_w == 1 && chroma_h == 0) { + return VPX_SUBSAMPLING_422; + } else if (chroma_w == 0 && chroma_h == 0) { + return VPX_SUBSAMPLING_444; + } + } + return -1; +} + +CFDataRef ff_videotoolbox_vpcc_extradata_create(AVCodecContext *avctx) +{ + const VP9SharedContext *h = avctx->priv_data; + CFDataRef data = NULL; + uint8_t *p; + int vt_extradata_size; + uint8_t *vt_extradata; + int subsampling = get_vpx_chroma_subsampling(avctx->sw_pix_fmt, avctx->chroma_sample_location); + + vt_extradata_size = 1 + 3 + 6 + 2; + vt_extradata = av_malloc(vt_extradata_size); + + if (subsampling < 0) + return NULL; + + if (!vt_extradata) + return NULL; + + p = vt_extradata; + + *p++ = 1; /* version */ + AV_WB24(p + 1, 0); /* flags */ + p += 3; + + *p++ = h->h.profile; + *p++ = avctx->level; + *p++ = (h->h.bpp << 4) | (subsampling << 1) | (avctx->color_range == AVCOL_RANGE_JPEG); + *p++ = avctx->color_primaries; + *p++ = avctx->color_trc; + *p++ = avctx->colorspace; + + AV_WB16(p + 0, 0); + p += 2; + + av_assert0(p - vt_extradata == vt_extradata_size); + + data = CFDataCreate(kCFAllocatorDefault, vt_extradata, vt_extradata_size); + av_free(vt_extradata); + return data; +} + +static int videotoolbox_vp9_start_frame(AVCodecContext *avctx, + const uint8_t *buffer, + uint32_t size) +{ + return 0; +} + +static int videotoolbox_vp9_decode_slice(AVCodecContext *avctx, + const uint8_t *buffer, + uint32_t size) +{ + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + + return ff_videotoolbox_buffer_copy(vtctx, buffer, size); +} + +static int videotoolbox_vp9_end_frame(AVCodecContext *avctx) +{ + const VP9SharedContext *h = avctx->priv_data; + AVFrame *frame = h->frames[CUR_FRAME].tf.f; + + return ff_videotoolbox_common_end_frame(avctx, frame); +} + +const AVHWAccel ff_vp9_videotoolbox_hwaccel = { + .name = "vp9_videotoolbox", + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_VP9, + .pix_fmt = AV_PIX_FMT_VIDEOTOOLBOX, + .alloc_frame = ff_videotoolbox_alloc_frame, + .start_frame = videotoolbox_vp9_start_frame, + .decode_slice = videotoolbox_vp9_decode_slice, + .end_frame = videotoolbox_vp9_end_frame, + .frame_params = ff_videotoolbox_frame_params, + .init = ff_videotoolbox_common_init, + .uninit = ff_videotoolbox_uninit, + .priv_data_size = sizeof(VTContext), +}; diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c index 8317ac6bd8..24841c5b9f 100644 --- a/libavcodec/vp9.c +++ b/libavcodec/vp9.c @@ -181,7 +181,8 @@ static int update_size(AVCodecContext *avctx, int w, int h) CONFIG_VP9_D3D11VA_HWACCEL * 2 + \ CONFIG_VP9_NVDEC_HWACCEL + \ CONFIG_VP9_VAAPI_HWACCEL + \ - CONFIG_VP9_VDPAU_HWACCEL) + CONFIG_VP9_VDPAU_HWACCEL + \ + CONFIG_VP9_VIDEOTOOLBOX_HWACCEL) enum AVPixelFormat pix_fmts[HWACCEL_MAX + 2], *fmtp = pix_fmts; VP9Context *s = avctx->priv_data; uint8_t *p; @@ -212,6 +213,9 @@ static int update_size(AVCodecContext *avctx, int w, int h) #endif #if CONFIG_VP9_VDPAU_HWACCEL *fmtp++ = AV_PIX_FMT_VDPAU; +#endif +#if CONFIG_VP9_VIDEOTOOLBOX_HWACCEL + *fmtp++ = AV_PIX_FMT_VIDEOTOOLBOX; #endif break; case AV_PIX_FMT_YUV420P12: @@ -1893,6 +1897,9 @@ const AVCodec ff_vp9_decoder = { #endif #if CONFIG_VP9_VDPAU_HWACCEL HWACCEL_VDPAU(vp9), +#endif +#if CONFIG_VP9_VIDEOTOOLBOX_HWACCEL + HWACCEL_VIDEOTOOLBOX(vp9), #endif NULL }, diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index e2f279d0fc..54a11fd1b5 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -64,5 +64,6 @@ int ff_videotoolbox_h264_decode_slice(AVCodecContext *avctx, int ff_videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame); CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx); CFDataRef ff_videotoolbox_hvcc_extradata_create(AVCodecContext *avctx); +CFDataRef ff_videotoolbox_vpcc_extradata_create(AVCodecContext *avctx); #endif /* AVCODEC_VT_INTERNAL_H */