From patchwork Wed Sep 27 01:08:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 5304 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp4455740jaa; Tue, 26 Sep 2017 18:08:34 -0700 (PDT) X-Received: by 10.223.174.175 with SMTP id y44mr9317493wrc.35.1506474514584; Tue, 26 Sep 2017 18:08:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506474514; cv=none; d=google.com; s=arc-20160816; b=J3jWVg3XdV9SP93fUffnLM/xIeDC9dUsoYdwciFhRdNuXz2WmFR+YMVA8HmOtNEO/r NHIDc24rO4xc+SPJAVJrvOYmjAdc3Ad8p1LCmUqEHUwYsVikvmF0gBr0fTMJI/BIyCpp GtR3vu2dCesQ37UZOMJ5knKQKNrMiKSz4/KvoLd3YgEPmXUu5UFNW8yZfbHBH/gKKwYz zgvyXyJvFDVJyF6yBRNFzshRY7Oei1HUwY+QvGSdPv5ws2MMF0rTzQSJRpfaJ10pPenR kHNIhcGH+thuG2wj9cK9/hCVSeLyaWqSKnv2/xtsXXiTYIrC3p+h6Lytr6hLZaUsj29V FxSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:delivered-to :arc-authentication-results; bh=th6Y+QlNRiaag4/ShcWlbW0dl6AfBGebD0ShbncWCMw=; b=oxJKfVkVMUT6MfnYu19+v+E9RTDKPE1LOjUAmqp8nmPWXr1O4B/2Ik4LgWFeLo+9mI +5SG6NAKkCrvUGC6WTQvVZ0wWQtJsgSBHKWm8cn+4QCMHCR3GTW6okwrKNhWUhFDK8MS bUvvMTTJBgaKmTtTFqaCrFmos3nuE8/fAFdYx8aNG5M7OP1bAiGckiRJ2z3WCHsQPkBl ISUUQh6EJ2PXnd8+HgTIdsfv6q7qwrmgBJVwMMU+7kEqzT8Yhdd/SsvqyQcS97y6Yb69 0azmIvCge+1XdU0vHWV+GNgy+16SCjp/Ps+ngf3xB+oYaPs+gEF+Wvd80xLqRAWyrx+i +MZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=PjUkUaF+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b9si8162409wrg.488.2017.09.26.18.08.34; Tue, 26 Sep 2017 18:08:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=PjUkUaF+; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2D1126883AE; Wed, 27 Sep 2017 04:08:15 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f52.google.com (mail-pg0-f52.google.com [74.125.83.52]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1E72A680284 for ; Wed, 27 Sep 2017 04:08:08 +0300 (EEST) Received: by mail-pg0-f52.google.com with SMTP id i130so6859519pgc.3 for ; Tue, 26 Sep 2017 18:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7xzcFdVfTl6uz3nZ18YeK8m4uN9zpFLWFeQJAi73HKQ=; b=PjUkUaF+6M48Ztl+4BhBlMWsC7swTINzZYRQwEDNbQCAqznvIm1atNl0cJhgdO+qeK n6nlEsIDSidi2zu4h03vO5oOJl6mr1weUwVSqHy3+B2graRdTJRpOfAvUR2aklU6m9TF s6g9Nc6DO8TapotSC4p1RstR76r3Et1zIW4QJQX4jq7b/Cch2EN/jIOHN1xaI3gpxWxc P9O0hC9PHqycNT6e2i97oHAMxHPZlRjeKuAm52PR/j4tjEk49QgVrTVyc07a9rCSwWhv y2hbzGW3i+nysNQs6INo/1cu76+0urxP+wInxYxZvql9FxXGtzuEg64m1CN2Vz0EVAVe Xq3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=7xzcFdVfTl6uz3nZ18YeK8m4uN9zpFLWFeQJAi73HKQ=; b=aX6tHGWZsBZm6t7G49jEY1BDUG00+lt26/O24RDejlzL17Vu0MF5bKb+9Y4JOHeo4V WIoZwZlpd5htQLul13MM6Ud55rh/DpqGbHApLgISQC95cnTMTjB3qATOw63sVRxAdTuT Som8nerwJwZ2YsGiMeda8GMtSYkjq3OleRHTJ+M4MLjRGy0pNpaAq25qCknrepu7vjaa gQfaErmE24ffbyAu8HQKXWPVIYZkO/LE81pc1W9i06MY2s0xFoyU82DlqCnZBkp9f34b 2JBGpPmFlxI3ieIdIzl3P/d39j5BNg4mxO8sRQKD5twimvIh2TwwhopISs/ca9/SJBnY TGow== X-Gm-Message-State: AHPjjUjsatCKxoTKEF1YyuFrQ2grGEV0urZBVWxZY7hvWBlYEkYuxnc7 eWN2m9FAvThS3LkoXAzxecTT/yr3 X-Google-Smtp-Source: AOwi7QDF3+RbAuBXXE52XZxFhndHf+4voI3DTD2gFcGET9G/ZDVaXTndTBztH+dlQI+PVlacZJ/A6g== X-Received: by 10.84.229.71 with SMTP id d7mr12083291pln.417.1506474497844; Tue, 26 Sep 2017 18:08:17 -0700 (PDT) Received: from tmm1-macbook.local.net (c-73-252-174-83.hsd1.ca.comcast.net. [73.252.174.83]) by smtp.gmail.com with ESMTPSA id e2sm13118260pfc.176.2017.09.26.18.08.17 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 26 Sep 2017 18:08:17 -0700 (PDT) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Tue, 26 Sep 2017 18:08:10 -0700 Message-Id: <20170927010810.66990-2-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.13.5 (Apple Git-94) In-Reply-To: <20170927010810.66990-1-ffmpeg@tmm1.net> References: <20170927010810.66990-1-ffmpeg@tmm1.net> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/2] avcodec/videotoolbox: add hevc support X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Aman Gupta Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta --- configure | 2 + libavcodec/allcodecs.c | 1 + libavcodec/hevc_refs.c | 3 + libavcodec/hevcdec.c | 12 ++- libavcodec/vda_vt_internal.h | 1 + libavcodec/videotoolbox.c | 203 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 221 insertions(+), 1 deletion(-) diff --git a/configure b/configure index ba38a73906..d353e9d824 100755 --- a/configure +++ b/configure @@ -2687,6 +2687,8 @@ hevc_vaapi_hwaccel_deps="vaapi VAPictureParameterBufferHEVC" hevc_vaapi_hwaccel_select="hevc_decoder" hevc_vdpau_hwaccel_deps="vdpau VdpPictureInfoHEVC" hevc_vdpau_hwaccel_select="hevc_decoder" +hevc_videotoolbox_hwaccel_deps="videotoolbox" +hevc_videotoolbox_hwaccel_select="hevc_decoder" mjpeg_cuvid_hwaccel_deps="cuda cuvid" mjpeg_cuvid_hwaccel_select="mjpeg_cuvid_decoder" mpeg_xvmc_hwaccel_deps="xvmc" diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c index db2db158f3..efc646634b 100644 --- a/libavcodec/allcodecs.c +++ b/libavcodec/allcodecs.c @@ -84,6 +84,7 @@ static void register_all(void) REGISTER_HWACCEL(HEVC_QSV, hevc_qsv); REGISTER_HWACCEL(HEVC_VAAPI, hevc_vaapi); REGISTER_HWACCEL(HEVC_VDPAU, hevc_vdpau); + REGISTER_HWACCEL(HEVC_VIDEOTOOLBOX, hevc_videotoolbox); REGISTER_HWACCEL(MJPEG_CUVID, mjpeg_cuvid); REGISTER_HWACCEL(MPEG1_CUVID, mpeg1_cuvid); REGISTER_HWACCEL(MPEG1_XVMC, mpeg1_xvmc); diff --git a/libavcodec/hevc_refs.c b/libavcodec/hevc_refs.c index 68c730edcc..ac462d350b 100644 --- a/libavcodec/hevc_refs.c +++ b/libavcodec/hevc_refs.c @@ -208,6 +208,9 @@ int ff_hevc_output_frame(HEVCContext *s, AVFrame *out, int flush) if (nb_output) { HEVCFrame *frame = &s->DPB[min_idx]; + if (frame->frame->format == AV_PIX_FMT_VIDEOTOOLBOX && frame->frame->buf[0]->size == 1) + return 0; + ret = av_frame_ref(out, frame->frame); if (frame->flags & HEVC_FRAME_FLAG_BUMPING) ff_hevc_unref_frame(s, frame, HEVC_FRAME_FLAG_OUTPUT | HEVC_FRAME_FLAG_BUMPING); diff --git a/libavcodec/hevcdec.c b/libavcodec/hevcdec.c index 2306c51ed3..2e4add2ae3 100644 --- a/libavcodec/hevcdec.c +++ b/libavcodec/hevcdec.c @@ -352,7 +352,11 @@ static void export_stream_params(AVCodecContext *avctx, const HEVCParamSets *ps, static enum AVPixelFormat get_format(HEVCContext *s, const HEVCSPS *sps) { - #define HWACCEL_MAX (CONFIG_HEVC_DXVA2_HWACCEL + CONFIG_HEVC_D3D11VA_HWACCEL * 2 + CONFIG_HEVC_VAAPI_HWACCEL + CONFIG_HEVC_VDPAU_HWACCEL) +#define HWACCEL_MAX (CONFIG_HEVC_DXVA2_HWACCEL + \ + CONFIG_HEVC_D3D11VA_HWACCEL * 2 + \ + CONFIG_HEVC_VAAPI_HWACCEL + \ + CONFIG_HEVC_VIDEOTOOLBOX_HWACCEL + \ + CONFIG_HEVC_VDPAU_HWACCEL) enum AVPixelFormat pix_fmts[HWACCEL_MAX + 2], *fmt = pix_fmts; switch (sps->pix_fmt) { @@ -371,6 +375,9 @@ static enum AVPixelFormat get_format(HEVCContext *s, const HEVCSPS *sps) #if CONFIG_HEVC_VDPAU_HWACCEL *fmt++ = AV_PIX_FMT_VDPAU; #endif +#if CONFIG_HEVC_VIDEOTOOLBOX_HWACCEL + *fmt++ = AV_PIX_FMT_VIDEOTOOLBOX; +#endif break; case AV_PIX_FMT_YUV420P10: #if CONFIG_HEVC_DXVA2_HWACCEL @@ -383,6 +390,9 @@ static enum AVPixelFormat get_format(HEVCContext *s, const HEVCSPS *sps) #if CONFIG_HEVC_VAAPI_HWACCEL *fmt++ = AV_PIX_FMT_VAAPI; #endif +#if CONFIG_HEVC_VIDEOTOOLBOX_HWACCEL + *fmt++ = AV_PIX_FMT_VIDEOTOOLBOX; +#endif break; } diff --git a/libavcodec/vda_vt_internal.h b/libavcodec/vda_vt_internal.h index e55a813899..326a60a695 100644 --- a/libavcodec/vda_vt_internal.h +++ b/libavcodec/vda_vt_internal.h @@ -59,4 +59,5 @@ int ff_videotoolbox_h264_decode_slice(AVCodecContext *avctx, const uint8_t *buffer, uint32_t size); CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx); +CFDataRef ff_videotoolbox_hvcc_extradata_create(AVCodecContext *avctx); #endif /* AVCODEC_VDA_VT_INTERNAL_H */ diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index dd13e2581b..078174cc61 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -32,6 +32,7 @@ #include "libavutil/hwcontext.h" #include "bytestream.h" #include "h264dec.h" +#include "hevcdec.h" #include "mpegvideo.h" #include @@ -115,6 +116,174 @@ CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx) return data; } +CFDataRef ff_videotoolbox_hvcc_extradata_create(AVCodecContext *avctx) +{ + HEVCContext *h = avctx->priv_data; + const HEVCVPS *vps = (const HEVCVPS *)h->ps.vps_list[0]->data; + const HEVCSPS *sps = (const HEVCSPS *)h->ps.sps_list[0]->data; + int i, num_pps = 0; + const HEVCPPS *pps = h->ps.pps; + PTLCommon ptlc = vps->ptl.general_ptl; + VUI vui = sps->vui; + uint8_t parallelismType; + CFDataRef data = NULL; + uint8_t *p; + int vt_extradata_size = 23 + 5 + vps->data_size + 5 + sps->data_size + 3; + uint8_t *vt_extradata; + + for (i = 0; i < MAX_PPS_COUNT; i++) { + if (h->ps.pps_list[i]) { + const HEVCPPS *pps = (const HEVCPPS *)h->ps.pps_list[i]->data; + vt_extradata_size += 2 + pps->data_size; + num_pps++; + } + } + + vt_extradata = av_malloc(vt_extradata_size); + if (!vt_extradata) + return NULL; + p = vt_extradata; + + /* unsigned int(8) configurationVersion = 1; */ + AV_W8(p + 0, 1); + + /* + * unsigned int(2) general_profile_space; + * unsigned int(1) general_tier_flag; + * unsigned int(5) general_profile_idc; + */ + AV_W8(p + 1, ptlc.profile_space << 6 | + ptlc.tier_flag << 5 | + ptlc.profile_idc); + + /* unsigned int(32) general_profile_compatibility_flags; */ + memcpy(p + 2, ptlc.profile_compatibility_flag, 4); + + /* unsigned int(48) general_constraint_indicator_flags; */ + AV_W8(p + 6, ptlc.progressive_source_flag << 7 | + ptlc.interlaced_source_flag << 6 | + ptlc.non_packed_constraint_flag << 5 | + ptlc.frame_only_constraint_flag << 4); + AV_W8(p + 7, 0); + AV_W8(p + 8, 0); + AV_W8(p + 9, 0); + AV_W8(p + 10, 0); + AV_W8(p + 11, 0); + + /* unsigned int(8) general_level_idc; */ + AV_W8(p + 12, ptlc.level_idc); + + /* + * bit(4) reserved = ‘1111’b; + * unsigned int(12) min_spatial_segmentation_idc; + */ + AV_W8(p + 13, 0xf0 | (vui.min_spatial_segmentation_idc >> 4)); + AV_W8(p + 14, vui.min_spatial_segmentation_idc & 0xff); + + /* + * bit(6) reserved = ‘111111’b; + * unsigned int(2) parallelismType; + */ + if (!vui.min_spatial_segmentation_idc) + parallelismType = 0; + else if (pps->entropy_coding_sync_enabled_flag && pps->tiles_enabled_flag) + parallelismType = 0; + else if (pps->entropy_coding_sync_enabled_flag) + parallelismType = 3; + else if (pps->tiles_enabled_flag) + parallelismType = 2; + else + parallelismType = 1; + AV_W8(p + 15, 0xfc | parallelismType); + + /* + * bit(6) reserved = ‘111111’b; + * unsigned int(2) chromaFormat; + */ + AV_W8(p + 16, sps->chroma_format_idc | 0xfc); + + /* + * bit(5) reserved = ‘11111’b; + * unsigned int(3) bitDepthLumaMinus8; + */ + AV_W8(p + 17, (sps->bit_depth - 8) | 0xfc); + + /* + * bit(5) reserved = ‘11111’b; + * unsigned int(3) bitDepthChromaMinus8; + */ + AV_W8(p + 18, (sps->bit_depth_chroma - 8) | 0xfc); + + /* bit(16) avgFrameRate; */ + AV_W8(p + 19, 0); + AV_W8(p + 20, 0); + + /* + * bit(2) constantFrameRate; + * bit(3) numTemporalLayers; + * bit(1) temporalIdNested; + * unsigned int(2) lengthSizeMinusOne; + */ + AV_W8(p + 21, 0 << 6 | + sps->max_sub_layers << 3 | + sps->temporal_id_nesting_flag << 2 | + 3); + + /* unsigned int(8) numOfArrays; */ + AV_W8(p + 22, 3); + + p += 23; + /* vps */ + /* + * bit(1) array_completeness; + * unsigned int(1) reserved = 0; + * unsigned int(6) NAL_unit_type; + */ + AV_W8(p, 1 << 7 | + HEVC_NAL_VPS & 0x3f); + /* unsigned int(16) numNalus; */ + AV_W8(p + 1, 0); + AV_W8(p + 2, 1); + /* unsigned int(16) nalUnitLength; */ + AV_W8(p + 3, vps->data_size >> 8); + AV_W8(p + 4, vps->data_size & 0xffff); + /* bit(8*nalUnitLength) nalUnit; */ + memcpy(p + 5, vps->data, vps->data_size); + p += 5 + vps->data_size; + + /* sps */ + AV_W8(p, 1 << 7 | + HEVC_NAL_SPS & 0x3f); + AV_W8(p + 1, 0); + AV_W8(p + 2, 1); + AV_W8(p + 3, sps->data_size >> 8); + AV_W8(p + 4, sps->data_size & 0xffff); + memcpy(p + 5, sps->data, sps->data_size); + p += 5 + sps->data_size; + + /* pps */ + AV_W8(p, 1 << 7 | + HEVC_NAL_PPS & 0x3f); + AV_W8(p + 1, num_pps >> 8); + AV_W8(p + 2, num_pps & 0xffff); + p += 3; + for (i = 0; i < MAX_PPS_COUNT; i++) { + if (h->ps.pps_list[i]) { + const HEVCPPS *pps = (const HEVCPPS *)h->ps.pps_list[i]->data; + AV_W8(p + 0, pps->data_size >> 8); + AV_W8(p + 1, pps->data_size & 0xffff); + memcpy(p + 2, pps->data, pps->data_size); + p += 2 + pps->data_size; + } + } + + av_assert0(p - vt_extradata == vt_extradata_size); + + data = CFDataCreate(kCFAllocatorDefault, vt_extradata, vt_extradata_size); + av_free(vt_extradata); + return data; +} + int ff_videotoolbox_buffer_create(VTContext *vtctx, AVFrame *frame) { av_buffer_unref(&frame->buf[0]); @@ -445,6 +614,18 @@ static int videotoolbox_h264_end_frame(AVCodecContext *avctx) return videotoolbox_common_end_frame(avctx, frame); } +static int videotoolbox_hevc_end_frame(AVCodecContext *avctx) +{ + HEVCContext *h = avctx->priv_data; + AVFrame *frame = h->ref->frame; + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + int ret; + + ret = videotoolbox_common_end_frame(avctx, frame); + vtctx->bitstream_size = 0; + return ret; +} + static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx, const uint8_t *buffer, uint32_t size) @@ -501,6 +682,11 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec if (data) CFDictionarySetValue(avc_info, CFSTR("avcC"), data); break; + case kCMVideoCodecType_HEVC : + data = ff_videotoolbox_hvcc_extradata_create(avctx); + if (data) + CFDictionarySetValue(avc_info, CFSTR("hvcC"), data); + break; default: break; } @@ -600,6 +786,9 @@ static int videotoolbox_default_init(AVCodecContext *avctx) case AV_CODEC_ID_H264 : videotoolbox->cm_codec_type = kCMVideoCodecType_H264; break; + case AV_CODEC_ID_HEVC : + videotoolbox->cm_codec_type = kCMVideoCodecType_HEVC; + break; case AV_CODEC_ID_MPEG1VIDEO : videotoolbox->cm_codec_type = kCMVideoCodecType_MPEG1Video; break; @@ -782,6 +971,20 @@ AVHWAccel ff_h263_videotoolbox_hwaccel = { .priv_data_size = sizeof(VTContext), }; +AVHWAccel ff_hevc_videotoolbox_hwaccel = { + .name = "hevc_videotoolbox", + .type = AVMEDIA_TYPE_VIDEO, + .id = AV_CODEC_ID_HEVC, + .pix_fmt = AV_PIX_FMT_VIDEOTOOLBOX, + .alloc_frame = ff_videotoolbox_alloc_frame, + .start_frame = ff_videotoolbox_h264_start_frame, + .decode_slice = ff_videotoolbox_h264_decode_slice, + .end_frame = videotoolbox_hevc_end_frame, + .init = videotoolbox_common_init, + .uninit = ff_videotoolbox_uninit, + .priv_data_size = sizeof(VTContext), +}; + AVHWAccel ff_h264_videotoolbox_hwaccel = { .name = "h264_videotoolbox", .type = AVMEDIA_TYPE_VIDEO,