From patchwork Fri Nov 10 21:40:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 5896 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.90 with SMTP id m26csp3884729jah; Fri, 10 Nov 2017 13:42:38 -0800 (PST) X-Google-Smtp-Source: AGs4zMZKjXHaMDc+SEZeGNSOdf6fqvVEWbFMfW5QEphV4hPXpTCSSSbt4WaLJxA8BagLDhNE7MZQ X-Received: by 10.28.145.13 with SMTP id t13mr1203144wmd.40.1510350158138; Fri, 10 Nov 2017 13:42:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510350158; cv=none; d=google.com; s=arc-20160816; b=KrZTw3HOxC9sXztUbPyu+4mrWKE7/YeyNZp4B1qpyKMtVtqrPASfvF0DVeLD9WgZbA N+t58UVMxLY/7bxHpo1IxCbtpgC8lrewrJRgqPuWqxV3SnjvPAsEz05jEwZ/4rtC9Irj w8+xLLw+n1IHjb02WrKud1jNY0Ue6SeoWyVAg7G70X896zZAJQ3MJJ+5dKBcs1pX7+OA M3e1iW3U90AXZM2vRRgcXQy/5SUyRqJ3Hr1O0Om3njqOjUCfeOX2Mu5jI0rGUNiB4RyF aagvC0JiXhKa5bAFbMw/xvFNpl8VmSSjgW+u7IE6LW4BhgoWHsuhV0SOviyJbXXed/dm fNcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=BtLPVCQ/i5gnxHsWVj/Ro93+z8QfOsEqgw4RwvF7LNA=; b=HTL/EyxLhBWwpJVlf/V9erxGkaHDlmQxRxRFyWBH2N8mShvyaWEalZLPrpJqYjzYvF jOeJ1BjX7ykla1Br8vOfbqg5y/biRtNIrMEsl3E7jfKUdtrdRfSS0337z8LFmGjQt2N2 6iFbMyMeT5y+DIZuxDn79etSNICpRxPSaJZFCk2Fqiyj+2aL3tjnyCwwY4d3ZJt09IDC wLT6ye07ewzW4IgDMBuDagyUz2M+dQX5IgesXBJadLI1/Im7eBj5FHKKt/x0tuNdX50W Lqw4KBfqqulk9VSggZ2deD1C5FBgEli8WywYrcfWOM+xoZee9HQ6brYFe3iRnYHcz4C8 yEqg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=L4SWNiNF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d36si9107806wrd.251.2017.11.10.13.42.37; Fri, 10 Nov 2017 13:42:38 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=L4SWNiNF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1C94D689F09; Fri, 10 Nov 2017 23:41:06 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pf0-f172.google.com (mail-pf0-f172.google.com [209.85.192.172]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 71A08689F17 for ; Fri, 10 Nov 2017 23:40:58 +0200 (EET) Received: by mail-pf0-f172.google.com with SMTP id a84so2639435pfl.0 for ; Fri, 10 Nov 2017 13:41:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=VwU2IJwcIleD44NtryuTMfskwfcWBz9G+eWBF+s8jl8=; b=L4SWNiNFENvuMFlePob5uZhD4mU+f/GPuRdAA4LIaEMntXR02WvIbXadEphKqcQ6wi YHNNw54w7ZTOYdpcJBHGqF2C9HFURpGLi/dWDsYlnyGA8swn9cfq8s4A//dnd7gScwQh M0qTNh2nS5483b6tazY3TWITF/2n2rOiO08tlJGNW6gIY8OFLcpLaBe3rASbGvbeav6y ErJmE3hzH7jsWT0p7tT0T9i1cP8jiNh1xlGoeF/xc5Pl0t9hfSyAKj2k/4r/+b+m4HiS +7fk8RXWm39d2fDmZo1Fua04NeNK/8In+0bZ+YCWliazsQv0wIMZwd9IGlXaG3eUt+Pj mxbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=VwU2IJwcIleD44NtryuTMfskwfcWBz9G+eWBF+s8jl8=; b=oQAQZ8hmkjHfgLc9kVEtLUrmsp+PLyVC/W3b3l6WkrAfhldN4i9F6r/s1NCbqcpQf5 TqKr6GXS1i2eLBS60o6vokKHYuiq4O9C/Jy8Def8C5ieU7asI4JGxNR7hO6DrJpG8Se/ FbVBuv02hUTAHZBcOCx/U/pRuM+18gvU2qWhnWxQwAvCsjkFzseFrKDEfeAJ1/FxMRE0 of5aLcy/T17Z6Ot9caMdIhvZO+VB9qizq98/YCSKx4yC66W2vTHDkX6CJ1JUF0wl5arj 098WVdnx6EJtrI7t3XqbKKF4uneEKJCgae5NXPk7Nmb9hFM3Z4bMFf/uLGLowkfRDcXf fxSg== X-Gm-Message-State: AJaThX4ITq7hIVR7WVUkoV1fu0TqCr3zryb8NuZae8G1DA1KX4515lx7 vK/NRAHg7+PSPtWwxEOnadO0LLA6 X-Received: by 10.99.119.73 with SMTP id s70mr1588960pgc.451.1510350070344; Fri, 10 Nov 2017 13:41:10 -0800 (PST) Received: from tmm1-imac.local.net (c-73-252-174-83.hsd1.ca.comcast.net. [73.252.174.83]) by smtp.gmail.com with ESMTPSA id 81sm20734554pfh.145.2017.11.10.13.41.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 10 Nov 2017 13:41:09 -0800 (PST) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Fri, 10 Nov 2017 13:40:53 -0800 Message-Id: <20171110214059.84891-8-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20171110214059.84891-1-ffmpeg@tmm1.net> References: <20171110214059.84891-1-ffmpeg@tmm1.net> Subject: [FFmpeg-devel] [PATCH v4 08/14] avcodec/videotoolbox: use decode_params to propagate H264 PPS changes and restart on SPS changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: nfxjfg@googlemail.com, Aman Gupta MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta This fixes decoding of H264 video samples with SPS and PPS changes. See for example https://s3.amazonaws.com/tmm1/videotoolbox/spschange.ts, which previously stalled the decoder and failed to produce any new frames after the SPS change. If the VideoToolbox session needs to be restarted, and videotoolbox_start() fails for some reason (for instance, if the video is interlaced and the decoder is running on iOS), avcodec will return AVERROR_EXTERNAL. This can be used by the API user to switch to another decoder. --- libavcodec/videotoolbox.c | 43 +++++++++++++++++++++++++++++++++++++++++-- libavcodec/vt_internal.h | 6 ++++++ 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 9db2e1116b..2496e4969c 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -42,6 +42,9 @@ enum { kCMVideoCodecType_HEVC = 'hvc1' }; #define VIDEOTOOLBOX_ESDS_EXTRADATA_PADDING 12 +static void videotoolbox_stop(AVCodecContext *avctx); +static int videotoolbox_start(AVCodecContext *avctx); + static void videotoolbox_buffer_release(void *opaque, uint8_t *data) { CVPixelBufferRef cv_buffer = (CVImageBufferRef)data; @@ -307,6 +310,27 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx, return 0; } +static int videotoolbox_h264_decode_params(AVCodecContext *avctx, + int type, + const uint8_t *buffer, + uint32_t size) +{ + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + + if (type == H264_NAL_SPS) { + if (!vtctx->sps || vtctx->sps_len != size || memcmp(buffer, vtctx->sps, size) != 0) { + vtctx->sps = av_fast_realloc(vtctx->sps, &vtctx->sps_capa, size); + if (vtctx->sps) + memcpy(vtctx->sps, buffer, size); + vtctx->reconfig_needed = true; + vtctx->sps_len = size; + } + } + + // pass-through new PPS to the decoder + return ff_videotoolbox_h264_decode_slice(avctx, buffer, size); +} + int ff_videotoolbox_h264_decode_slice(AVCodecContext *avctx, const uint8_t *buffer, uint32_t size) @@ -339,6 +363,7 @@ int ff_videotoolbox_uninit(AVCodecContext *avctx) VTContext *vtctx = avctx->internal->hwaccel_priv_data; if (vtctx) { av_freep(&vtctx->bitstream); + av_freep(&vtctx->sps); if (vtctx->frame) CVPixelBufferRelease(vtctx->frame); } @@ -591,17 +616,30 @@ static int videotoolbox_common_end_frame(AVCodecContext *avctx, AVFrame *frame) AVVideotoolboxContext *videotoolbox = videotoolbox_get_context(avctx); VTContext *vtctx = avctx->internal->hwaccel_priv_data; - if (!videotoolbox->session || !vtctx->bitstream) + if (vtctx->reconfig_needed == true) { + vtctx->reconfig_needed = false; + av_log(avctx, AV_LOG_VERBOSE, "VideoToolbox decoder needs reconfig, restarting..\n"); + videotoolbox_stop(avctx); + if (videotoolbox_start(avctx) != 0) { + return AVERROR_EXTERNAL; + } + } + + if (!videotoolbox->session || !vtctx->bitstream || !vtctx->bitstream_size) return AVERROR_INVALIDDATA; status = videotoolbox_session_decode_frame(avctx); if (status != noErr) { + if (status == kVTVideoDecoderMalfunctionErr || status == kVTInvalidSessionErr) + vtctx->reconfig_needed = true; av_log(avctx, AV_LOG_ERROR, "Failed to decode frame (%s, %d)\n", videotoolbox_error_string(status), (int)status); return AVERROR_UNKNOWN; } - if (!vtctx->frame) + if (!vtctx->frame) { + vtctx->reconfig_needed = true; return AVERROR_UNKNOWN; + } return videotoolbox_buffer_create(avctx, frame); } @@ -1003,6 +1041,7 @@ AVHWAccel ff_h264_videotoolbox_hwaccel = { .alloc_frame = ff_videotoolbox_alloc_frame, .start_frame = ff_videotoolbox_h264_start_frame, .decode_slice = ff_videotoolbox_h264_decode_slice, + .decode_params = videotoolbox_h264_decode_params, .end_frame = videotoolbox_h264_end_frame, .init = videotoolbox_common_init, .uninit = videotoolbox_uninit, diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index 607dd7a6d0..fc27dad9f9 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -38,6 +38,12 @@ typedef struct VTContext { // Non-NULL if the new hwaccel API is used. This is only a separate struct // to ease compatibility with the old API. struct AVVideotoolboxContext *vt_ctx; + + // Current H264 parameters (used to trigger decoder restart on SPS changes). + uint8_t *sps; + uint32_t sps_len; + unsigned int sps_capa; + bool reconfig_needed; } VTContext; int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame);