From patchwork Mon Nov 22 21:58:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: rcombs X-Patchwork-Id: 31583 Delivered-To: andriy.gelman@gmail.com Received: by 2002:ab0:6957:0:0:0:0:0 with SMTP id c23csp7338264uas; Mon, 22 Nov 2021 13:59:41 -0800 (PST) X-Google-Smtp-Source: ABdhPJy+YfNzK6uBi5EXps0HPLvlcyjxGU6tkLLl9528brCCUGhyvfoEL8atooZ1qs9GPPEXa06P X-Received: by 2002:a05:6402:11d2:: with SMTP id j18mr619142edw.318.1637618381297; Mon, 22 Nov 2021 13:59:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1637618381; cv=none; d=google.com; s=arc-20160816; b=qsCs1QPZUXHRNfEGA8A9Uh+55LVb79mOECr5/ZUNJm67jF9HMtk5xckyCC85ilyzKV 60D+v1vQtPMGOrAnm7WmN97EVYC8v9TSUQoyvup2H23ziapkGh+7KVgKlEL2gxFnjOah KzTNKFmersg2DgRZqKo+Sb43hWLUw8NP0j3YE8t90IchyRlhDJOQ5yUosH0rvKSu0oKA fLxtygOZwpoYXUYFig3LAC1b24Y2UCpj8gGxTvq8zbYQ9jmpNvxGirQbk5XST1dXqxHy uW1W/5O3Px25FYpJfkKsMvH+k4GijyyvqSXqCxzVkLbbIzE54QOfG2e9x6piXBZthAFD LGWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:references:in-reply-to:message-id :date:to:from:dkim-signature:delivered-to; bh=eZ0qzurqRe37Owxk4Ow8IVlTgPWA+WCJXGZp+DhF6+w=; b=pOYMbc7U7QbZ0Zm5HatYOYPeM3LoYQ6s9h3SEhgRZuNVcopGzH/y0NxRU16+FgiWiT jQZO5eJ3FZ0yg1K5HypE8soVaKEsyKFuW7yLJoMQGt2ypBmzLDrIirZ+Z8QhmDx0mHCz dXXKk4gjCsF/hcZAyoEYTE663PvW74JiSNcFAKd+iJ3kGci8+7ZkNTPADeYuhv+WXXX3 sTcq1pVP6GPW2S04AcV77uzzyWZHApRslufotjBmzJNBxfAqimaLzGLzRFoAAW5wHtNE DlOkbBv4QRe5Vgh+pdYFmqa2s/DmEtaiJgydGyqTQBgTu+2sw13u/XatJYfH6k65GGKZ AiXg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=Nse8EM9u; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id hs10si28354907ejc.712.2021.11.22.13.59.40; Mon, 22 Nov 2021 13:59:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@rcombs.me header.s=google header.b=Nse8EM9u; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=rcombs.me Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1424468AC75; Mon, 22 Nov 2021 23:58:45 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 16A5168AA75 for ; Mon, 22 Nov 2021 23:58:42 +0200 (EET) Received: by mail-io1-f49.google.com with SMTP id e144so25292610iof.3 for ; Mon, 22 Nov 2021 13:58:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rcombs.me; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=S4tY74O1rWCxl5cTsOhSoStbjporJFcND2PPBXSAWto=; b=Nse8EM9uqwXykIZYWx4oM4VgEgYDd3+nyriyp8SQctkrkaozbfPTLEueuHT99wAcxb 0q7qKKiaCePFGPIRM/JlhifI7UrtOs1uK7joLEFFInS5mmkFVUger8gJPNCsJimDQqJQ 2dd9GvnfbFPwiOa7RgvnZqyknSgCqhUNlQDhtW9VaMP53O+PS+ia22CK+UrWaRHLNCxd EakfAhffJSGpxJxUz+pv50mHhltG/N3nyFtTfmg8oXWnNTO8FuINPIhn9eWcfsO0kydl Pr2t6v3++m6ezw6W+R95w9aR+H7OkVwjvjmOqQKLT02snvJtX6jKMfq5xR1HANd0HViY dRhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S4tY74O1rWCxl5cTsOhSoStbjporJFcND2PPBXSAWto=; b=H2fMmsR07aQCCosphMym/oT1dR+3j7mbA51LfME2nX3wSRbtdwufwTGCICpJ1ZorVR jQUvZ5x89/757pmlp4RHSqMIEQVAjBGhWo0SwY3y4SGnuDnpbiZaZHIHs6LvUXVRwk80 RafTlwuOZ9Zo+wbp/9qh4dyXEvjIQGLLO2AinN9GJ5lKMlqp/LmhpZ+HnTWCxnsZYCHq meYGpLjSAwxWW4SNmx4hdrIYQ1gPp9NEw544wAbPwUj/W3gsJu3Z6EUbxgJ1VCCJjMCN nEz8QWzPEqjL0614mgTeUf1ws27HiMug+Zf3yJn7uGq+LKZ+Z74H0kKOYsdBzH3M5sa7 /wAA== X-Gm-Message-State: AOAM530ALG/pOoXyU8T5n7lB6xLVCZRDv/A37uQHrSzAWETG0i97P6OA ZdFccwisp8//b9Dr4XPXkQwtHl9vjhFESuY= X-Received: by 2002:a02:a314:: with SMTP id q20mr341082jai.104.1637618321013; Mon, 22 Nov 2021 13:58:41 -0800 (PST) Received: from rcombs-mbp.localdomain ([2601:243:2000:5ac:edf5:499:2479:42a8]) by smtp.gmail.com with ESMTPSA id o1sm5668802ilj.41.2021.11.22.13.58.39 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Nov 2021 13:58:40 -0800 (PST) From: rcombs To: ffmpeg-devel@ffmpeg.org Date: Mon, 22 Nov 2021 15:58:17 -0600 Message-Id: <20211122215821.9849-12-rcombs@rcombs.me> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211122215821.9849-1-rcombs@rcombs.me> References: <20211122215821.9849-1-rcombs@rcombs.me> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 12/16] lavc/videotoolbox: add VP9 hardware acceleration X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: A4rwS6KqGKRv Content-Length: 12276 On M1 Max, this supports profiles 0 and 2, but not 1 and 3. --- Changelog | 1 + configure | 4 + libavcodec/Makefile | 1 + libavcodec/hwaccels.h | 1 + libavcodec/videotoolbox.c | 17 ++++- libavcodec/videotoolbox_vp9.c | 140 ++++++++++++++++++++++++++++++++++ libavcodec/vp9.c | 9 ++- libavcodec/vt_internal.h | 1 + 8 files changed, 171 insertions(+), 3 deletions(-) create mode 100644 libavcodec/videotoolbox_vp9.c diff --git a/Changelog b/Changelog index 31a0d5ef5d..d6be6bbe4f 100644 --- a/Changelog +++ b/Changelog @@ -31,6 +31,7 @@ version : - varblur video filter - huesaturation video filter - colorspectrum source video filter +- VideoToolbox VP9 hwaccel version 4.4: diff --git a/configure b/configure index cc79b5b793..262261b544 100755 --- a/configure +++ b/configure @@ -2328,6 +2328,7 @@ TOOLCHAIN_FEATURES=" TYPES_LIST=" kCMVideoCodecType_HEVC kCMVideoCodecType_HEVCWithAlpha + kCMVideoCodecType_VP9 kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange @@ -3083,6 +3084,8 @@ vp9_vaapi_hwaccel_deps="vaapi VADecPictureParameterBufferVP9_bit_depth" vp9_vaapi_hwaccel_select="vp9_decoder" vp9_vdpau_hwaccel_deps="vdpau VdpPictureInfoVP9" vp9_vdpau_hwaccel_select="vp9_decoder" +vp9_videotoolbox_hwaccel_deps="videotoolbox" +vp9_videotoolbox_hwaccel_select="vp9_decoder" wmv3_d3d11va_hwaccel_select="vc1_d3d11va_hwaccel" wmv3_d3d11va2_hwaccel_select="vc1_d3d11va2_hwaccel" wmv3_dxva2_hwaccel_select="vc1_dxva2_hwaccel" @@ -6287,6 +6290,7 @@ enabled videotoolbox && { check_lib coreservices CoreServices/CoreServices.h UTGetOSTypeFromString "-framework CoreServices" check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_HEVC "-framework CoreMedia" check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_HEVCWithAlpha "-framework CoreMedia" + check_func_headers CoreMedia/CMFormatDescription.h kCMVideoCodecType_VP9 "-framework CoreMedia" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange "-framework CoreVideo" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_422YpCbCr8BiPlanarVideoRange "-framework CoreVideo" check_func_headers CoreVideo/CVPixelBuffer.h kCVPixelFormatType_422YpCbCr10BiPlanarVideoRange "-framework CoreVideo" diff --git a/libavcodec/Makefile b/libavcodec/Makefile index 14fbd2ecbc..32b35ebe75 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -975,6 +975,7 @@ OBJS-$(CONFIG_VP9_DXVA2_HWACCEL) += dxva2_vp9.o OBJS-$(CONFIG_VP9_NVDEC_HWACCEL) += nvdec_vp9.o OBJS-$(CONFIG_VP9_VAAPI_HWACCEL) += vaapi_vp9.o OBJS-$(CONFIG_VP9_VDPAU_HWACCEL) += vdpau_vp9.o +OBJS-$(CONFIG_VP9_VIDEOTOOLBOX_HWACCEL) += videotoolbox_vp9.o OBJS-$(CONFIG_VP8_QSV_HWACCEL) += qsvdec.o # libavformat dependencies diff --git a/libavcodec/hwaccels.h b/libavcodec/hwaccels.h index 8e54cf73f9..65e778f3e4 100644 --- a/libavcodec/hwaccels.h +++ b/libavcodec/hwaccels.h @@ -74,6 +74,7 @@ extern const AVHWAccel ff_vp9_dxva2_hwaccel; extern const AVHWAccel ff_vp9_nvdec_hwaccel; extern const AVHWAccel ff_vp9_vaapi_hwaccel; extern const AVHWAccel ff_vp9_vdpau_hwaccel; +extern const AVHWAccel ff_vp9_videotoolbox_hwaccel; extern const AVHWAccel ff_wmv3_d3d11va_hwaccel; extern const AVHWAccel ff_wmv3_d3d11va2_hwaccel; extern const AVHWAccel ff_wmv3_dxva2_hwaccel; diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index a11d494915..5f08488a6b 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -33,6 +33,7 @@ #include "hevcdec.h" #include "mpegvideo.h" #include +#include #include #ifndef kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder @@ -46,6 +47,10 @@ enum { kCMVideoCodecType_HEVC = 'hvc1' }; #endif +#if !HAVE_KCMVIDEOCODECTYPE_VP9 +enum { kCMVideoCodecType_VP9 = 'vp09' }; +#endif + #define VIDEOTOOLBOX_ESDS_EXTRADATA_PADDING 12 typedef struct VTHWFrame { @@ -816,6 +821,11 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec if (data) CFDictionarySetValue(avc_info, CFSTR("hvcC"), data); break; + case kCMVideoCodecType_VP9 : + data = ff_videotoolbox_vpcc_extradata_create(avctx); + if (data) + CFDictionarySetValue(avc_info, CFSTR("vpcC"), data); + break; default: break; } @@ -863,12 +873,15 @@ static int videotoolbox_start(AVCodecContext *avctx) case AV_CODEC_ID_MPEG4 : videotoolbox->cm_codec_type = kCMVideoCodecType_MPEG4Video; break; + case AV_CODEC_ID_VP9 : + videotoolbox->cm_codec_type = kCMVideoCodecType_VP9; + break; default : break; } -#ifdef __MAC_10_11 - if (__builtin_available(macOS 10.11, *)) { +#if defined(MAC_OS_VERSION_11_0) && !TARGET_OS_IPHONE && (MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_VERSION_11_0) + if (__builtin_available(macOS 11.0, *)) { VTRegisterSupplementalVideoDecoderIfAvailable(videotoolbox->cm_codec_type); } #endif diff --git a/libavcodec/videotoolbox_vp9.c b/libavcodec/videotoolbox_vp9.c new file mode 100644 index 0000000000..1b6d08f00b --- /dev/null +++ b/libavcodec/videotoolbox_vp9.c @@ -0,0 +1,140 @@ +/* + * Videotoolbox hardware acceleration for VP9 + * + * copyright (c) 2021 rcombs + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "config.h" +#include "videotoolbox.h" +#include "libavutil/hwcontext_videotoolbox.h" +#include "vt_internal.h" +#include "libavutil/avutil.h" +#include "libavutil/frame.h" +#include "libavutil/hwcontext.h" +#include "libavutil/intreadwrite.h" +#include "libavutil/pixdesc.h" +#include "decode.h" +#include "internal.h" +#include "vp9shared.h" + +enum VPX_CHROMA_SUBSAMPLING +{ + VPX_SUBSAMPLING_420_VERTICAL = 0, + VPX_SUBSAMPLING_420_COLLOCATED_WITH_LUMA = 1, + VPX_SUBSAMPLING_422 = 2, + VPX_SUBSAMPLING_444 = 3, +}; + +static int get_vpx_chroma_subsampling(enum AVPixelFormat pixel_format, + enum AVChromaLocation chroma_location) +{ + int chroma_w, chroma_h; + if (av_pix_fmt_get_chroma_sub_sample(pixel_format, &chroma_w, &chroma_h) == 0) { + if (chroma_w == 1 && chroma_h == 1) { + return (chroma_location == AVCHROMA_LOC_LEFT) + ? VPX_SUBSAMPLING_420_VERTICAL + : VPX_SUBSAMPLING_420_COLLOCATED_WITH_LUMA; + } else if (chroma_w == 1 && chroma_h == 0) { + return VPX_SUBSAMPLING_422; + } else if (chroma_w == 0 && chroma_h == 0) { + return VPX_SUBSAMPLING_444; + } + } + return -1; +} + +CFDataRef ff_videotoolbox_vpcc_extradata_create(AVCodecContext *avctx) +{ + const VP9SharedContext *h = avctx->priv_data; + CFDataRef data = NULL; + uint8_t *p; + int vt_extradata_size; + uint8_t *vt_extradata; + int subsampling = get_vpx_chroma_subsampling(avctx->sw_pix_fmt, avctx->chroma_sample_location); + + vt_extradata_size = 1 + 3 + 6 + 2; + vt_extradata = av_malloc(vt_extradata_size); + + if (subsampling < 0) + return NULL; + + if (!vt_extradata) + return NULL; + + p = vt_extradata; + + *p++ = 1; /* version */ + AV_WB24(p + 1, 0); /* flags */ + p += 3; + + *p++ = h->h.profile; + *p++ = avctx->level; + *p++ = (h->h.bpp << 4) | (subsampling << 1) | (avctx->color_range == AVCOL_RANGE_JPEG); + *p++ = avctx->color_primaries; + *p++ = avctx->color_trc; + *p++ = avctx->colorspace; + + AV_WB16(p + 0, 0); + p += 2; + + av_assert0(p - vt_extradata == vt_extradata_size); + + data = CFDataCreate(kCFAllocatorDefault, vt_extradata, vt_extradata_size); + av_free(vt_extradata); + return data; +} + +static int videotoolbox_vp9_start_frame(AVCodecContext *avctx, + const uint8_t *buffer, + uint32_t size) +{ + return 0; +} + +static int videotoolbox_vp9_decode_slice(AVCodecContext *avctx, + const uint8_t *buffer, + uint32_t size) +{ + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + + return ff_videotoolbox_buffer_copy(vtctx, buffer, size); +} + +static int videotoolbox_vp9_end_frame(AVCodecContext *avctx) +{ + const VP9SharedContext *h = avctx->priv_data; + AVFrame *frame = h->frames[CUR_FRAME].tf.f; + + return ff_videotoolbox_common_end_frame(avctx, frame); +} + +const AVHWAccel ff_vp9_videotoolbox_hwaccel = { + .name = "vp9_videotoolbox", + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_VP9, + .pix_fmt = AV_PIX_FMT_VIDEOTOOLBOX, + .alloc_frame = ff_videotoolbox_alloc_frame, + .start_frame = videotoolbox_vp9_start_frame, + .decode_slice = videotoolbox_vp9_decode_slice, + .end_frame = videotoolbox_vp9_end_frame, + .frame_params = ff_videotoolbox_frame_params, + .init = ff_videotoolbox_common_init, + .uninit = ff_videotoolbox_uninit, + .priv_data_size = sizeof(VTContext), +}; diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c index 8317ac6bd8..24841c5b9f 100644 --- a/libavcodec/vp9.c +++ b/libavcodec/vp9.c @@ -181,7 +181,8 @@ static int update_size(AVCodecContext *avctx, int w, int h) CONFIG_VP9_D3D11VA_HWACCEL * 2 + \ CONFIG_VP9_NVDEC_HWACCEL + \ CONFIG_VP9_VAAPI_HWACCEL + \ - CONFIG_VP9_VDPAU_HWACCEL) + CONFIG_VP9_VDPAU_HWACCEL + \ + CONFIG_VP9_VIDEOTOOLBOX_HWACCEL) enum AVPixelFormat pix_fmts[HWACCEL_MAX + 2], *fmtp = pix_fmts; VP9Context *s = avctx->priv_data; uint8_t *p; @@ -212,6 +213,9 @@ static int update_size(AVCodecContext *avctx, int w, int h) #endif #if CONFIG_VP9_VDPAU_HWACCEL *fmtp++ = AV_PIX_FMT_VDPAU; +#endif +#if CONFIG_VP9_VIDEOTOOLBOX_HWACCEL + *fmtp++ = AV_PIX_FMT_VIDEOTOOLBOX; #endif break; case AV_PIX_FMT_YUV420P12: @@ -1893,6 +1897,9 @@ const AVCodec ff_vp9_decoder = { #endif #if CONFIG_VP9_VDPAU_HWACCEL HWACCEL_VDPAU(vp9), +#endif +#if CONFIG_VP9_VIDEOTOOLBOX_HWACCEL + HWACCEL_VIDEOTOOLBOX(vp9), #endif NULL }, diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index e2f279d0fc..54a11fd1b5 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -64,5 +64,6 @@ int ff_videotoolbox_h264_decode_slice(AVCodecContext *avctx, int ff_videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame); CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx); CFDataRef ff_videotoolbox_hvcc_extradata_create(AVCodecContext *avctx); +CFDataRef ff_videotoolbox_vpcc_extradata_create(AVCodecContext *avctx); #endif /* AVCODEC_VT_INTERNAL_H */