From patchwork Wed Nov 15 21:15:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 6106 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.161.94 with SMTP id m30csp5054915jah; Wed, 15 Nov 2017 13:15:41 -0800 (PST) X-Google-Smtp-Source: AGs4zMaTgSzH6o+PDRLwO1OmSRP+MGAK/NY81XF7Iabm5KNz4ci7gc0ToNg9vLK24VYDSSWgV+ix X-Received: by 10.28.237.18 with SMTP id l18mr11923416wmh.136.1510780541676; Wed, 15 Nov 2017 13:15:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510780541; cv=none; d=google.com; s=arc-20160816; b=rfVxsCxuvgCRixAkOn9tMUNpShVYA/riC+pJtNmAnPokjwJbX8AWrXqgJc7U9UrP1Q nhgkLzMrLXGQkKsq95sxWCuiGfuQFZjm5ZAxJ36EBKdT0+QPc/eqjSzL+I0uK4XQ2Rq9 ZNkaku0alkKhUXSH66WP/LcbHqAkmcANJ2GuijCGAIH6ncUwmn9JX0xIZF0+h0VAACx+ 7BgoRgCapaQPVD/o8vGn38CnR8nHxnp25L9mMM4rnM8CtgYBp/6+goa6Qp3aNDu65Hvc nk1Lw9etzJy1cANP8iqPfSCG3oAu5ANBkce6RYdSZvpfefZe7Im7r4oMbuSoaQu2Elxq xpoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=s6+9KNBaH3x1Gm4UQ2tshIh5ND+aChGePPW2fPbS3GA=; b=kdxDL+ZvnYHrbqhoaVUAJyl7F+Ki62e5y5CMJXdoHMmFgC7ZgwEE3v8NeYKsTSN5Sn Zq0mJn1N50x3wvvbN+U63kHcmQodAtL5KenJonuHyBrWDkoBc9t1+qB7u5/Gse3jKjTn poTdDEF4VEJBhLSBULDfpF6S9VT+UY6Lk4HKIHXNo/D4QAla72QvN8S9nQvzsK8oLMdH tr/w6SxK7v3b7TH+/ih+FcF5AoiMUMY8wNstGWl4fzgiG1xW9qeirG2B8Y+VAJY11CWD T5ok0Iqfyt+SuqyG8i8V+7eeafT/TZp28MTKnzIOkYpCO9yY6Pw86n2yjY/qQF7TsYTA lViw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=sIPL6MM7; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id d23si18450355wrb.552.2017.11.15.13.15.40; Wed, 15 Nov 2017 13:15:41 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=sIPL6MM7; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2492368A031; Wed, 15 Nov 2017 23:15:24 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-qt0-f195.google.com (mail-qt0-f195.google.com [209.85.216.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EA8BD689C8F for ; Wed, 15 Nov 2017 23:15:16 +0200 (EET) Received: by mail-qt0-f195.google.com with SMTP id f8so37697897qta.5 for ; Wed, 15 Nov 2017 13:15:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id; bh=mVjcHTbZrLPRSbrP0qUzGamsquj1C94GXg5eZB3BDfU=; b=sIPL6MM7DGdIEZ6LzkQhtyRR72w7QFVP3zronBnWd7rl3EJ+rW+oK9saXsmJCKCr0A qh64IPRYP/rmf50tbTBMwXf1uzkWEzIkexv5bopof6vbKJOM4lh8uZIwRdIzV1/LYub0 SLOw48SPEqMMB7QnVmiU2/MsUdlseLrc2r/j3LtlGKavvDs19VkAf0ZbRFQUVQW6+OcR SpDTqPlee3Vd2oyjxgU7MVY9Vu46E8aL7S4d80PrBXpUuEAqOF9G6e04rKdu2MLiAxZ8 JSe6YGLhH+f13xHbZQwNMqWEoPGObJP20vvKMochO8XrApMzqcwuSJtbJDBHl/Ronht8 eFrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id; bh=mVjcHTbZrLPRSbrP0qUzGamsquj1C94GXg5eZB3BDfU=; b=k5M3eRdss4CbArckc2F5EOihxfBJ7KX7cNRLHK4B+DCdU9vfKUudzSnx6g6GFmcO6L Ah9IjcyLyeDZmPT21ckH8+GvNQ4DUeWJNIAjM1yF4fGb1SpTPcN5K1QWrBHUxYwc7hth en3cYsIU9N4fkEQIiGwl/iHOKuNx2nBjkoqtGJ2ozNyt21uPFesJKPAHbeojVwhiqsPY BkYVaKO+cv/2oobF72QWhJn3EZzvFIbsEu+GcL97a16lh3lSLTbnbFQsc3HD+xu6eMIf Ec1NkpvuKVP1Y4sQMBh7+263K1QQtcdGE/ywyytrJaHNuW3UOFDhkeWqj5qCnXgQTCuK DaEA== X-Gm-Message-State: AJaThX6il8QpUUJn3VuS0XZ9iAVac79K4acZ8q9CXyYlOpHYb42VKvsv yfd2LeWlr7pCf9n8t4Cgh3BWiVCe X-Received: by 10.55.195.216 with SMTP id r85mr25088609qkl.218.1510780530896; Wed, 15 Nov 2017 13:15:30 -0800 (PST) Received: from tmm1-imac.local.net (c-24-6-8-87.hsd1.ca.comcast.net. [24.6.8.87]) by smtp.gmail.com with ESMTPSA id q12sm1045015qtk.32.2017.11.15.13.15.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 15 Nov 2017 13:15:30 -0800 (PST) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Wed, 15 Nov 2017 13:15:06 -0800 Message-Id: <20171115211506.78836-1-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.14.2 Subject: [FFmpeg-devel] [PATCH] libavcodec/videotoolbox: fix decoding of h264 streams with minor SPS changes X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: nfxjfg@googlemail.com, Aman Gupta , kernrj@gmail.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta Previously the codec kept an entire copy of the SPS, and restarted the VT decoder session whenever it changed. This fixed decoding errors in [1], as described in 9519983c. On further inspection, that sample features an SPS change from High/4.0 to High/3.2 while moving from one scene to another. Yesterday I received [2], which contains minor SPS changes where the profile and level do not change. These occur frequently and are not associated with scene changes. After 9519983c, the VT decoder session is recreated unnecessarily when these are encountered causing visual glitches. This commit simplifies the state kept in the VTContext to include just the first three bytes of the SPS, containing the profile and level details. This is populated initially when the VT decoder session is created, and used to detect changes and force a restart. This means minor SPS changes are fed directly into the existing decoder, whereas profile/level changes force the decoder session to be recreated with the new parameters. After this commit, both samples [1] and [2] playback as expected. [1] https://s3.amazonaws.com/tmm1/videotoolbox/spschange.ts [2] https://s3.amazonaws.com/tmm1/videotoolbox/spschange2.ts Signed-off-by: Aman Gupta --- libavcodec/videotoolbox.c | 15 ++++++++------- libavcodec/vt_internal.h | 4 +--- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 9eeada30ba..d29607363c 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -90,6 +90,7 @@ int ff_videotoolbox_alloc_frame(AVCodecContext *avctx, AVFrame *frame) CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx) { + VTContext *vtctx = avctx->internal->hwaccel_priv_data; H264Context *h = avctx->priv_data; CFDataRef data = NULL; uint8_t *p; @@ -116,6 +117,10 @@ CFDataRef ff_videotoolbox_avcc_extradata_create(AVCodecContext *avctx) p += 3 + h->ps.pps->data_size; av_assert0(p - vt_extradata == vt_extradata_size); + // save sps header (profile/level) used to create decoder session, + // so we can detect changes and recreate it. + memcpy(vtctx->sps, h->ps.sps->data + 1, 3); + data = CFDataCreate(kCFAllocatorDefault, vt_extradata, vt_extradata_size); av_free(vt_extradata); return data; @@ -320,16 +325,13 @@ static int videotoolbox_h264_decode_params(AVCodecContext *avctx, VTContext *vtctx = avctx->internal->hwaccel_priv_data; if (type == H264_NAL_SPS) { - if (!vtctx->sps || vtctx->sps_len != size || memcmp(buffer, vtctx->sps, size) != 0) { - vtctx->sps = av_fast_realloc(vtctx->sps, &vtctx->sps_capa, size); - if (vtctx->sps) - memcpy(vtctx->sps, buffer, size); + if (size > 4 && memcmp(vtctx->sps, buffer + 1, 3) != 0) { vtctx->reconfig_needed = true; - vtctx->sps_len = size; + memcpy(vtctx->sps, buffer + 1, 3); } } - // pass-through new PPS to the decoder + // pass-through SPS/PPS changes to the decoder return ff_videotoolbox_h264_decode_slice(avctx, buffer, size); } @@ -365,7 +367,6 @@ int ff_videotoolbox_uninit(AVCodecContext *avctx) VTContext *vtctx = avctx->internal->hwaccel_priv_data; if (vtctx) { av_freep(&vtctx->bitstream); - av_freep(&vtctx->sps); if (vtctx->frame) CVPixelBufferRelease(vtctx->frame); } diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index fc27dad9f9..929fe423fc 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -40,9 +40,7 @@ typedef struct VTContext { struct AVVideotoolboxContext *vt_ctx; // Current H264 parameters (used to trigger decoder restart on SPS changes). - uint8_t *sps; - uint32_t sps_len; - unsigned int sps_capa; + uint8_t sps[3]; bool reconfig_needed; } VTContext;