From patchwork Tue Sep 26 00:36:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 5277 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp3231671jaa; Mon, 25 Sep 2017 17:45:25 -0700 (PDT) X-Received: by 10.28.230.198 with SMTP id e67mr1523286wmi.8.1506386725203; Mon, 25 Sep 2017 17:45:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506386725; cv=none; d=google.com; s=arc-20160816; b=ObQC6oLKGtIgIYPbVzQjmoBvfnmPjYBnLOyCRJKsMCJqYTa++yJzpw24B1v9/ROGQX ZX/EK3sbC+qwaHTy5tkmKDwbWCdtWLGeX7HfJaAGMjtpVSZ02wQMnNtnN2Je0xWsE9ln +fG0dp4qLtDMYMgQRuih+JM8qqooNgXXJLqARXx97Nv6zK+SGhkQ+FoMJ849goJDvRfW a9/3U4jzVJ7O7WbKYfuGrkRU6N1hx72D2PQJVLp3ltmBbyAXGXejipzt2qEslh6s8XWS Ol8aYkFWd3WWoukgUVPFoi+bLpw9Uw/XiBMpmOohBOv9cXdadBMEfs/D5ArlWrZpaWQS VB1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=ly6DjEPpc56uKD82QRkzKeq5U8jK37oQvZTYdoF/sXg=; b=ifZXC07li67Q95euE2ZefHgDrw4TcJQ4UMLDck5y06Qw2tDXoi43f2IApcbdrlTSas 2CvpDIeUU1E514abrXt1MCW8OhQdlg++tkPDz/txpkNwJg8Pr+4q5ay56epUOwiNaROu 4e2vdDhrQ6cr1JrkxibFuxw0IdKTIpRAKXqmNQmub7A7YY6wnceyrOWZ1YfeXJ617eR9 bR6nznE7TehJs/bKsWPuzzZjATlCKFAWRCjKTxdUhbdgGbVUo9jPd0XIvIYDIPX1G3Sm 1m9/j/pHW7F+dD+MS5FU97yc2KqQTQSCXnJ7dgmE2JXXSTi7f+MV0d3c1Akqi51NIdVg Hnng== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=T+DHc5Vd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b13si5955498wrh.491.2017.09.25.17.45.24; Mon, 25 Sep 2017 17:45:25 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=T+DHc5Vd; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 80F5E689B41; Tue, 26 Sep 2017 03:45:12 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f54.google.com (mail-pg0-f54.google.com [74.125.83.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id E51A4689AFC for ; Tue, 26 Sep 2017 03:45:05 +0300 (EEST) Received: by mail-pg0-f54.google.com with SMTP id 7so4976488pgd.13 for ; Mon, 25 Sep 2017 17:45:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ux6xOsSYR9+WX5xIK3n3YxjA08uXDeaYpUXxqM3IQ9c=; b=T+DHc5VdqwwdUsB87Gniuu2ZzRbTHHUz/i6TAw8CIfMCZh+E7qh1BZy8IxzY3SZ93C Axj8ynmQ4vbrGyuLMYEVbN26TUpyhi3ExShtAg4bRBPPm3qaCllRmYfRqAZhjZtPqVdF IFVb/FoVeA5/FxtzyL81H5txUt8GI5x7dnTb/MLG2TrwXoZD7eNrdjhAIv50Zmjuc+EP SkAMBNG3vH58C41sIT4lup/2AY6jbTVkHxfKYrnpczmFb3Py2dpb0cAQ/IGy5mc4nvau FkYmm5KIvT+lV7W72Cm1zMg4+bAnh5MqphqK/lZ1sal5wrv8r8ZTPrDD8t2mI16zKkc7 tjBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=Ux6xOsSYR9+WX5xIK3n3YxjA08uXDeaYpUXxqM3IQ9c=; b=Co/rj8neGAEr0PnTp9TUTMgq/uLttfbpuyt7troQiHu9BgA2eq/kQHH+S9gVZVI69a CLuOZi6aaV/jhpK/W7heKHwdrhUHfz5mjebhPXq5Vo7P/2MP15uiO/G2t/mxT1oQOmAZ YFltVX5qG+m+VlQh25zD+PO386nMRvX3VkpB1jdrBM1wcqxdpdfMN/6khy9XPU2ZJaPt NTEeKyxbLvAl+RIY9VRy4iNWg2akxHCB+yNFqXwMmi2qd7E/X5fSGinVvnj7aSZgRXw6 bN3VN0MXmSD92YDQBzi+zeGbynIZ6ulzHIEAM5l2XXBZBBagck2Hk5jupPn2Lf7h8kSd xT6Q== X-Gm-Message-State: AHPjjUgSSCyyP45wiIKG0d0Iq3fb5Z0q23zjpFkQ49OYs66rnWp7zRsB gA7oBSxvluLjCNt91mK+OBkRMbj5 X-Google-Smtp-Source: AOwi7QD7/x3evlzCGeTi9yhnv2ZEaeJl+J8+sUE4sGAusBUjSEUhZViA2VDrlLE+GZcnQnFZnv47Hg== X-Received: by 10.84.248.138 with SMTP id q10mr9229734pll.431.1506386210478; Mon, 25 Sep 2017 17:36:50 -0700 (PDT) Received: from tmm1-macbook.local.net (c-73-252-174-83.hsd1.ca.comcast.net. [73.252.174.83]) by smtp.gmail.com with ESMTPSA id n18sm12483164pgd.69.2017.09.25.17.36.49 (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 25 Sep 2017 17:36:50 -0700 (PDT) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Mon, 25 Sep 2017 17:36:30 -0700 Message-Id: <20170926003631.55571-8-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.13.5 (Apple Git-94) In-Reply-To: <20170926003631.55571-1-ffmpeg@tmm1.net> References: <20170926003631.55571-1-ffmpeg@tmm1.net> Subject: [FFmpeg-devel] [PATCH 8/9] avcodec/videotoolbox: use decode_params to propagate PPS changes and restart on SPS changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Aman Gupta MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta If the VideoToolbox session needs to be restarted, and videotoolbox_start() fails for some reason (for instance, if the video is interlaced and the decoder is running on iOS), avcodec will return AVERROR_EXTERNAL. This can be used by the API user to switch to another decoder. --- libavcodec/vda_vt_internal.h | 6 ++++++ libavcodec/videotoolbox.c | 45 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/libavcodec/vda_vt_internal.h b/libavcodec/vda_vt_internal.h index e55a813899..49c91791ee 100644 --- a/libavcodec/vda_vt_internal.h +++ b/libavcodec/vda_vt_internal.h @@ -47,6 +47,12 @@ typedef struct VTContext { // Non-NULL if the new hwaccel API is used. This is only a separate struct // to ease compatibility with the old API. struct AVVideotoolboxContext *vt_ctx; + + // Current H264 parameters (used to trigger decoder restart on SPS changes). + uint8_t *sps; + uint32_t sps_len; + unsigned int sps_capa; + bool reconfig_needed; } VTContext; int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame); diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index f56ab1f8c9..6c8477c2ce 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -41,6 +41,9 @@ #define VIDEOTOOLBOX_ESDS_EXTRADATA_PADDING 12 +static void videotoolbox_stop(AVCodecContext *avctx); +static int videotoolbox_start(AVCodecContext *avctx); + static void videotoolbox_buffer_release(void *opaque, uint8_t *data) { CVPixelBufferRef cv_buffer = (CVImageBufferRef)data; @@ -148,6 +151,33 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx, return 0; } +static int videotoolbox_h264_decode_params(AVCodecContext *avctx, + const uint8_t *buffer, + uint32_t size) +{ + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + H264Context *h = avctx->priv_data; + + if (h->is_avc == 1) + return 0; + + switch (buffer[0] & 0x1f) { + case H264_NAL_SPS: + if (!vtctx->sps || vtctx->sps_len != size || memcmp(buffer, vtctx->sps, size) != 0) { + vtctx->sps = av_fast_realloc(vtctx->sps, &vtctx->sps_capa, size); + if (vtctx->sps) + memcpy(vtctx->sps, buffer, size); + if (vtctx->sps_len) + vtctx->reconfig_needed = true; + vtctx->sps_len = size; + } + break; + } + + // pass-through new PPS to the decoder + return ff_videotoolbox_h264_decode_slice(avctx, buffer, size); +} + int ff_videotoolbox_h264_decode_slice(AVCodecContext *avctx, const uint8_t *buffer, uint32_t size) @@ -180,6 +210,7 @@ int ff_videotoolbox_uninit(AVCodecContext *avctx) VTContext *vtctx = avctx->internal->hwaccel_priv_data; if (vtctx) { av_freep(&vtctx->bitstream); + av_freep(&vtctx->sps); if (vtctx->frame) CVPixelBufferRelease(vtctx->frame); } @@ -419,7 +450,16 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame) AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx); VTContext *vtctx = avctx->internal->hwaccel_priv_data; - if (!videotoolbox->session || !vtctx->bitstream) + if (vtctx->reconfig_needed == true) { + vtctx->reconfig_needed = false; + av_log(avctx, AV_LOG_VERBOSE, "VideoToolbox decoder needs reconfig, restarting..\n"); + videotoolbox_stop(avctx); + if (videotoolbox_start(avctx) != 0) { + return AVERROR_EXTERNAL; + } + } + + if (!videotoolbox->session || !vtctx->bitstream || !vtctx->bitstream_size) return AVERROR_INVALIDDATA; status = videotoolbox_session_decode_frame(avctx); @@ -432,9 +472,11 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame) break; case kVTVideoDecoderMalfunctionErr: error = "decoder malfunction"; + vtctx->reconfig_needed = true; break; case kVTInvalidSessionErr: error = "invalid session"; + vtctx->reconfig_needed = true; break; default: error = "unknown"; @@ -824,6 +866,7 @@ AVHWAccel ff_h264_videotoolbox_hwaccel = { .alloc_frame = ff_videotoolbox_alloc_frame, .start_frame = ff_videotoolbox_h264_start_frame, .decode_slice = ff_videotoolbox_h264_decode_slice, + .decode_params = videotoolbox_h264_decode_params, .end_frame = videotoolbox_h264_end_frame, .init = videotoolbox_common_init, .uninit = videotoolbox_uninit,