From patchwork Tue Sep 26 00:36:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 5271 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp3229918jaa; Mon, 25 Sep 2017 17:42:53 -0700 (PDT) X-Received: by 10.28.229.147 with SMTP id c141mr1415473wmh.156.1506386573558; Mon, 25 Sep 2017 17:42:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506386573; cv=none; d=google.com; s=arc-20160816; b=Qkqdrn/sKvYemm/LUaYHtg9DGOos8dXF/WTnHAicXZZo6onr6vzU30sbUqXd9jLK8i QtWkpZ1J3UtwFmjMdpd11nvRvWccUcCQfqx9eAKmwOvHIPHUso+LCwxFcy5eoJCZk6dj aROiF+QYVfShRvuRrHfSe9eikz4OSOL97uvu2YB1y00OC3wTlyYxvIbqWHSjCcYQrjKI kTtxrI8tm4qDFHspxECatrj/zmW9hn8xlRquXh3X6cskTdwUEiF0bTaTlIK9cr6+66Q8 WTLtLFow/+znhU+m8rfDUnh/r+fT7atFMcWMC/DxVRJoIj+7Ktdkpj8mn7y3UwgnwfDL bRYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=70d2PL/BzrhL/1qX+6LfGt0ZuWy+B1p4p7722xqTLNo=; b=lxdKIkg0P9B/7cG9KYfZ94nehZXy3x0YSpyOxLOE/dgL58N3/t0OwDcaPBrxQGzPP8 eBcFDBxT217ZWOdoJN4C8KltKgwKi2yEZiXf54rjQlhrXDjpqV9RtoSr7P/Lu6q1A9og lF9ypbGSiC34MGbCReBxdLUVeHVKktgI/2sOeTwRwW9ayVtWeErIzo78NtASfLUlTb+O sEa7x1k8wKm7jUyYy5xPfnZHKXU3MAdQ/mEMySgp0NqNbqTU+H9zCbtbeTrvx0ER9Nc3 uTS9TH+nJrW7PXJ7fomVGiJ6EuPtgNPM8b1Nz6fTC1TYxt1x+4CWl1k3I/FVbmVFC9w8 WF8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=Wov7vBgl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id j77si488694wmj.265.2017.09.25.17.42.53; Mon, 25 Sep 2017 17:42:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=Wov7vBgl; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AFD5D689A8B; Tue, 26 Sep 2017 03:42:40 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg0-f68.google.com (mail-pg0-f68.google.com [74.125.83.68]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 667696805A7 for ; Tue, 26 Sep 2017 03:42:33 +0300 (EEST) Received: by mail-pg0-f68.google.com with SMTP id j16so5777288pga.2 for ; Mon, 25 Sep 2017 17:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=iVi83tTdvHE1b2fPcL5f3NwzCK3qHlPaPYMLXiWoJPo=; b=Wov7vBglosNk31/f1zaZYaKNsyaofBGTBNLRlxwTHpH+36MrktoXCloJd4Qlvq7lEi +mi3/b3FfUv7TLLUsk88PiJGExJ4o/ZmpVcc6IYMeTRU9CXphry8mKeeZ0FqXfo/Xe2T RGYPtiqqtKlH+CKNIdxHYcq8JRPOh3/QY6VbTSBPqlTKTvj7L5MQMTTopJrokWlVAAuJ ayB8k0rK3/IjMDFlkbO8OwtC2DNPlmEF/prOo8gHa9C3vfUtJCn5AhfKESVgM9u4z0wv iif56EJ5S/Xgf/DY3JSVQEDdhO8Ufr5kGnLKymYP8Pw06Sv6Of6XrSAjz1YqFVzQBln2 4z1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=iVi83tTdvHE1b2fPcL5f3NwzCK3qHlPaPYMLXiWoJPo=; b=bOHaWdr+DJaWpo2K7zdW9MuFmWPD3V/98lfFQ+T1EFiBxWjY1V0h0O4wIO1/+FoFEp XIpcU7sdYxtb36aXJwcPEJptrHPC2GYeWfIOp4F/Vr3E51bFL1ETBtr6PylieLDrE8+b 2WW8c0iKx8Num7pnZ1vVOIh/M4OaigN5OBB1cPXjiJ2h8XPL1mSyzW2XYhnURsamO7mE NcUQsYtZnscaxRXsuIzhi8UpQVfUQ2P8Rki+lLpx05BcEjiIXmoyvkv5q5lJ667tc6D6 FIqVakzf9w+dopkFyiXyAOIQm3S2U70bYVWt+4l3CgZcs7asmHGdASpz9ESGn1oD6h/q n5vA== X-Gm-Message-State: AHPjjUgg2pbGoqGCjAI7k2FRj9lxWBbtWKLTCaz59HSl/k1LKWHmJX5B wH+SALz3iLctbPNzhT0p6h+TczFj X-Google-Smtp-Source: AOwi7QBphvaJtfxGOtVDAfNEsi0wgtwqwwKdhNnETsL27xlt1Z+Srrtpfsiv7rpFC0KkvBxEcE0G1Q== X-Received: by 10.84.131.67 with SMTP id 61mr9138809pld.418.1506386205678; Mon, 25 Sep 2017 17:36:45 -0700 (PDT) Received: from tmm1-macbook.local.net (c-73-252-174-83.hsd1.ca.comcast.net. [73.252.174.83]) by smtp.gmail.com with ESMTPSA id n18sm12483164pgd.69.2017.09.25.17.36.44 (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 25 Sep 2017 17:36:45 -0700 (PDT) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Mon, 25 Sep 2017 17:36:25 -0700 Message-Id: <20170926003631.55571-3-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.13.5 (Apple Git-94) In-Reply-To: <20170926003631.55571-1-ffmpeg@tmm1.net> References: <20170926003631.55571-1-ffmpeg@tmm1.net> Subject: [FFmpeg-devel] [PATCH 3/9] avcodec/videotoolbox: h264 decoder on iOS is unable to decode interlaced video X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Aman Gupta MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta VideoToolbox's support for interlaced H264 is quite poor. On macOS, VTSessionCopySupportedPropertyDictionary() will show that kVTDecompressionPropertyKey_FieldMode is supported. Possible values for this option include DeinterlaceFields and BothFields. However, files that use MBAFF interlacing are not deinterlaced on macOS even if the FieldMode=DeinterlacedFields option is specified. Although it doesn't always deinterlace, the macOS version of VideoToolbox will always decode and return frame data even when the H264 source is interlaced. On iOS, FieldMode is not a valid option and interlaced H264 is not supported at all. You can create a valid decompression session, but no frames are returned and almost every DecodeFrame() call returns a malfunction error. I opened rdar://30669495 about this, and Apple's response was: > Correct and intentional. Please stop using interlaced video. So this commit forces the VideoToolbox hwaccel to fail early when interlaced H264 is encountered. Thus the API user can easily detect the failure and use a different decoder. Here are some sample files I tested on iOS 10 and 11: https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts https://s3.amazonaws.com/tmm1/videotoolbox/interlaced2.ts https://s3.amazonaws.com/tmm1/videotoolbox/interlaced3.ts Decoding interlaced2.ts with VideoToolbox never produces any frames. Decoding interlaced3.ts ocassional produces frames, but most slices cause errors. Decoding interlaced.ts actually works as expected, even though its SPS matches the other two samples which fail. This means my test is not comprehensive, and it is not possible to fully detect if a file is compatible with VideoToolbox using its SPS alone. Still, though this method produces false positives, it does not produce false negatives. --- libavcodec/videotoolbox.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 7dfcf14c00..8ed56392cc 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -496,11 +496,18 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec if (data) CFDictionarySetValue(avc_info, CFSTR("esds"), data); break; - case kCMVideoCodecType_H264 : + case kCMVideoCodecType_H264 : { + H264Context *h = avctx->priv_data; + if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0) { + av_log(avctx, AV_LOG_ERROR, "VideoToolbox cannot decode interlaced fields on iOS\n"); + CFRelease(avc_info); + goto fail; + } data = ff_videotoolbox_avcc_extradata_create(avctx); if (data) CFDictionarySetValue(avc_info, CFSTR("avcC"), data); break; + } default: break; } @@ -515,6 +522,10 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec CFRelease(avc_info); } return config_info; + +fail: + CFRelease(config_info); + return NULL; } static CFDictionaryRef videotoolbox_buffer_attributes_create(int width,