From patchwork Tue Aug 13 06:44:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: sharpbai X-Patchwork-Id: 14478 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id B1875449248 for ; Tue, 13 Aug 2019 09:50:02 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8974368AB93; Tue, 13 Aug 2019 09:50:02 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1A71E68AB83 for ; Tue, 13 Aug 2019 09:49:56 +0300 (EEST) Received: by mail-pl1-f194.google.com with SMTP id m9so48709432pls.8 for ; Mon, 12 Aug 2019 23:49:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dFzc92aahjU8dl1c1Mt1rmvP6fZkK2y7SIRRniyP66A=; b=kQ69bsNfOUa+KoKXfyrhJOjTe12HtQx32IoTXatL4dmpYsMMfiq/oqdAV5BGLpOJ5l YBv6KoSEvPQzWeqmfnOmbTDYhcM9oD2RyJNIk/MI87ENhT7G2iwLRoPT10o/AUNkCdV9 Dd+iLw0kpSimBw6LOodo3JWvrEbDNmqXpG/h9ZflGnB9W1ipn3c7083qGJCrrJ+hgM8L uG/xCEMWGJgVexC90Px2Rk1EDX6zAIVHOUymzBLd66iPcBcBukuFUT1deI6MdTUcGVgu C23jLkgN/jC8OylJpU0usBl7i7f1cUJd2foTR1bK3rnhUMPF4NuphmnGecOUa5AVkUih jRkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dFzc92aahjU8dl1c1Mt1rmvP6fZkK2y7SIRRniyP66A=; b=KagHahqRY+OeehdvkxNTKoV7FRN2FuKNsipi6YF1YViAKXhoMvV0l77jV7YCgUCEjj tJ8WhWEpCEM3v8oPLhUqab9PeMZRDBVmmsMe+7DHygX9ByspwW07Rxts7Fbzo2LpG7DB 0hM7XRFWnGBWjYMXPQgVqSUimF9gGRzT/Crafe4itRaXZuAqclZxxQ/m47Yk/5gLaKqF 7bYLdkzDmoInuvgd5TDOsX2MYJm3E8vmT1pJpbPT500J/qE0njcL9jYx+Pd8oYhmKhbg sZR1H3gSDkRDhfBaT8pIxTVBR2c9i2qwBM1Yy7oqIumdqJIt/gfBq8lk6PQSdaPuWgqX fugQ== X-Gm-Message-State: APjAAAVyCM81zU/QdyJIFlvJsxJOUXD1zqRFIhnzlACDHlJlIW+Lshva Y+gPngT/+OlzGenRiERZqkvu6LQBRCc= X-Google-Smtp-Source: APXvYqx0QvXv1EWqs+xhcurAkTKVMFNoI2okGbG7i7mOWV/QjqBE6KIwJxxRisuzLhB9qW6AJf0Leg== X-Received: by 2002:a17:902:9a49:: with SMTP id x9mr37043867plv.282.1565678663780; Mon, 12 Aug 2019 23:44:23 -0700 (PDT) Received: from localhost.localdomain (ec2-13-230-214-69.ap-northeast-1.compute.amazonaws.com. [13.230.214.69]) by smtp.gmail.com with ESMTPSA id z63sm82956386pfb.98.2019.08.12.23.44.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 12 Aug 2019 23:44:23 -0700 (PDT) From: sharpbai To: ffmpeg-devel@ffmpeg.org Date: Tue, 13 Aug 2019 14:44:02 +0800 Message-Id: <20190813064401.59869-1-sharpbai@gmail.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: fix encoding frame crash on iOS 11 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: sharpbai Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On iOS 11, encoding a frame may return error with log "Error encoding frame 0", which means vtenc_output_callback is called with status=0 and sample_buffer=NULL. Then the encoding session will be crashed on next callback wether or not closing the codec context. Let us look through the link below introducing VTCompressionOutputCallback, https://developer.apple.com/documentation/videotoolbox/vtcompressionoutputcallback?language=objc "status=0" (noErr) means compression was successful. "sampleBuffer=NULL" means the frame was dropped when compression was successful (status=0) or compression was not successful (status!=0). So we should not set AVERROR_EXTERNAL on "status=0" and "sample_buffer=NULL" as it is not a error. The fix is that we only set AVERROR_EXTERNAL with status value non zero. When sample_buffer is NULL and status value is zero, we simply return with no other operation. This crash often occurs on iOS 11 for example encoding 720p@25fps. Signed-off-by: sharpbai --- libavcodec/videotoolboxenc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index d76bb7f646..8afdd125d2 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -569,12 +569,16 @@ static void vtenc_output_callback( return; } - if (status || !sample_buffer) { + if (status) { av_log(avctx, AV_LOG_ERROR, "Error encoding frame: %d\n", (int)status); set_async_error(vtctx, AVERROR_EXTERNAL); return; } + if (!sample_buffer) { + return; + } + if (!avctx->extradata && (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER)) { int set_status = set_extradata(avctx, sample_buffer); if (set_status) {