From patchwork Tue Jul 31 21:56:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Karmani X-Patchwork-Id: 9853 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp112966jad; Tue, 31 Jul 2018 15:03:21 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdn7ohqj0rhrp54wp+9sAj8xCbiK7fNaCuEuXBQN483Y50BslWs6EnNeOHvgmoBEnsrAnop X-Received: by 2002:a1c:e455:: with SMTP id b82-v6mr978021wmh.93.1533074601775; Tue, 31 Jul 2018 15:03:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533074601; cv=none; d=google.com; s=arc-20160816; b=k0FrNW+y+DP8W5h/0ipI11FzoalEm4gIlsPhjUrBQeyeOC+VTB7+P2DH0NFi2eCICe THVqC+YhS7K7AlhFmBOxkcEyqCjJ1scjAW0dWVh9fInbtEFmffM6RR47mguo98kfV5st 0R6tX+9qL/63VL/TJ8ohNTfp2EhyqknXHqPo/PegWcwgb52PfHDe4uT/SC6Wuk5K3a4X P8iRZcHX4E7NPFi+jg4ntvGNuf5ncEUbSX6mo6N2XF4M4hNwJAPjnCZgZxspDfipG0io mqNoLfdnAXw+87SJbNhMdM/K15nigh39hg7r42MkIvQ/DxNyoGnlqtQ9TYJNpxjaNDUh l0QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=Zx/uunCJ7271byjxf/BtG91hzPoqKfZz2bMHSkcYAFk=; b=AfQUbRrUitAHlid1pMPlyVaeyNgFWcxb9slFunM2fwtLT8WgR0QL4NflLY+pOmEMxI 1/Bnhv6+7rOD789NEL0pHIthmfQgdoMyt/jF7FftW2g9qs2wCWzzC1rSCDB6bfdWd4cA Mk/RoZ9Pwam82Qnvzt3nqrNX6Xgj1mi5nrmiCFhpx6whe4yDr469g18HOzKgHoDQ7DQS PUygpLXry28nDUPpZpnj8ExpoW77apq8pwQ361yy/dmDN+scChQo9qyM8/I+63wRvtmi YfC6z7zKIjYHiRzKM7agDuk6DnQfp7YnsAAYDe5U9o3x+up0Ly0xSfiDCxWb2CFD27PJ bTJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=vvtHNkfc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 135-v6si2103258wmk.55.2018.07.31.15.03.20; Tue, 31 Jul 2018 15:03:21 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@tmm1-net.20150623.gappssmtp.com header.s=20150623 header.b=vvtHNkfc; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 5602868A456; Wed, 1 Aug 2018 01:03:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A5E7068A38C for ; Wed, 1 Aug 2018 01:02:55 +0300 (EEST) Received: by mail-pg1-f196.google.com with SMTP id z8-v6so9732245pgu.8 for ; Tue, 31 Jul 2018 15:03:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm1-net.20150623.gappssmtp.com; s=20150623; h=sender:from:to:cc:subject:date:message-id; bh=EBmAufq3ThY0+Z5AkBYpvFzX+QTriHC4v8A+Q5e8VJY=; b=vvtHNkfcUpmXjSZCvpbVxeLlTPshU/4VQnxSKlwz/dEvkZoF2Ud/QO5qHZaw2rE46r 6JCKccqw0Ej3JDwxHGOjNYLoVJYfgjJk+P9kuy0ItzCI/z4zybSPqXDasHe46KDR1KLi T1yeQFhAqsndwPN0fI3AZJFKcV+vh2s3SYrfER6m1veG81KloW1ai2IOSUBaWrPjqiEo 33tNaRSdbCvZ/C76pDDGPXRGBdRoUdW1pZNe022Wlzm3sLO7qv6/842UlckoD30/GqCc dTk4M5obLfgvDmYf1Ed7rqVp3GstgFnNdlEQKL7lXUhjBuNu4d4mIEtebhiWMtoXktAP sxSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id; bh=EBmAufq3ThY0+Z5AkBYpvFzX+QTriHC4v8A+Q5e8VJY=; b=Rq5bTkJ54XYJJi2Aot0A685Ql4hrPPU58+oMnmXCX8eQy8AOe7V6GJx0myGkKhOen5 /JFuYv5w0YM0YJOwzn+AnOgVolaPEgWLwmQx+jWPOum2DdY4JlZVx/gsVjExZJKCKrqu nA/0yzt8shmnMOhA/hrwxdX5XE4in97Id6LIcvF9HWDgyucHyG2jHL/9r2WMEPskAm5V oE7oEc7esdjhgrPsD3RarSj8OCZBB8iCs44SMJDelB5s35vWfmPm668H/9PW8Xi+fibU S2KTnrXRxjJGAOXwkh5v6vc/vxzaSw1CeaQ6TlwdBPdCJ1adgs4Iqckob5W0SR+S0n3O dAHg== X-Gm-Message-State: AOUpUlGIR6RmrolszG+03Hz+mXQ/xA6YgpvFek6qt0EU5Soo4S6VDkOP KMxmuIMZgcEhVac4xsxHnUvsyc/rYiI= X-Received: by 2002:a65:6102:: with SMTP id z2-v6mr22328394pgu.46.1533074188686; Tue, 31 Jul 2018 14:56:28 -0700 (PDT) Received: from localhost.localdomain (c-69-181-54-242.hsd1.ca.comcast.net. [69.181.54.242]) by smtp.gmail.com with ESMTPSA id d12-v6sm20109723pfn.118.2018.07.31.14.56.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 31 Jul 2018 14:56:28 -0700 (PDT) From: Aman Gupta To: ffmpeg-devel@ffmpeg.org Date: Tue, 31 Jul 2018 14:56:11 -0700 Message-Id: <20180731215611.2027-1-ffmpeg@tmm1.net> X-Mailer: git-send-email 2.18.0 Subject: [FFmpeg-devel] [PATCH] avcodec/mediacodecdec: fix SEGV on modern nvidia decoders X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: matthieu.bouron@gmail.com, Aman Gupta , pb.mythtv@gmail.com MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Aman Gupta This code came originally from gstreamer, where it was added in [1] as a work-around for the Tegra 3. (The alignment was changed in [2] as a response to [3], from 32-bit to 16-bit). gstreamer only used this workaround in the case where the decoder didn't return a slice-height property, but when the code was copied into avcodec the conditional got lost. This commit restores the guard and prefers the slice-height from the decoder when it is available. This fixes segfaults decoding 1920x1080 h264 and mpeg2 videos on the NVidia SHIELD after upgrading to Android Oreo. [1] https://github.com/GStreamer/gst-plugins-bad/commit/a870e6a5c30dd85240fe75c7409cc1cf1b86541d [2] https://github.com/GStreamer/gst-plugins-bad/commit/21ff3ae0b0127bd82951d278ca24f2d54133b7cd [3] https://bugzilla.gnome.org/show_bug.cgi?id=748867 Signed-off-by: Aman Gupta --- libavcodec/mediacodecdec_common.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/libavcodec/mediacodecdec_common.c b/libavcodec/mediacodecdec_common.c index 887865a281..f235dae71b 100644 --- a/libavcodec/mediacodecdec_common.c +++ b/libavcodec/mediacodecdec_common.c @@ -389,13 +389,14 @@ static int mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecConte s->stride = s->stride > 0 ? s->stride : s->width; AMEDIAFORMAT_GET_INT32(s->slice_height, "slice-height", 0); - s->slice_height = s->slice_height > 0 ? s->slice_height : s->height; - if (strstr(s->codec_name, "OMX.Nvidia.")) { + if (strstr(s->codec_name, "OMX.Nvidia.") && s->slice_height == 0) { s->slice_height = FFALIGN(s->height, 16); } else if (strstr(s->codec_name, "OMX.SEC.avc.dec")) { s->slice_height = avctx->height; s->stride = avctx->width; + } else if (s->slice_height == 0) { + s->slice_height = s->height; } AMEDIAFORMAT_GET_INT32(s->color_format, "color-format", 1);