From patchwork Tue Jul 3 15:05:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Guillem X-Patchwork-Id: 9594 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp1225081jad; Tue, 3 Jul 2018 08:06:09 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdqkZUq2Lk8Ql/n6jYT310ZTQ9U4J4uiZuQhXS5CHq0hKkYfui+kV/I4JIyIWO+AVnwEAmW X-Received: by 2002:adf:9736:: with SMTP id r51-v6mr22438783wrb.5.1530630369616; Tue, 03 Jul 2018 08:06:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530630369; cv=none; d=google.com; s=arc-20160816; b=iqz304PboL2qe3wMVrwKUyRkTJcinVckAg96+LhrNPmRC8QI7wZ+oNIUq4YMraY8x4 t7weBrAIZHhpHVRlCzrhrquyLdHTkG/YqM5f+ptTeVGY1Hxl2xruahGWEKSXTB4KcI6+ +qONVZvb7ZqMOiDBPxgUqptH9ydL3ixXLXm5UOirYOJSB15pPFQRe6ZOVSGgNv94PzEQ lZ8IzCLHHCPSM5bE2v+lWOJ4s8Y0Kz2T/tKJ/+++5d/yJbKMpWhmVQZ25oKv4P9rloT9 BZe8W/v/S0u235TWdzvBMLGaX2nlGXzWj7ZsDJtgK2zdCW5JP0gHKUsmur10xxDM5xn8 gldg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :dkim-signature:delivered-to:arc-authentication-results; bh=dQTmXav5o/8R2UQnpZ9G1C77UvsgZorX5GlKP2vcXT0=; b=sEQwuZj3qrM8sJANPZ8QOgfihgovOOWl/X7bIOWEqtN95dtqVv4reXn7B2p6mldQFR 3ivx01tEOCeXQejASDp00AEbaSAYXnslT44WDoBbQEaLDdh2iJoaMAU/4Dd42shS/1xi YZFYMLqIBdM/yMax1smBZG9lmvrpr2QA48Ag2BaBbYp96NWyfSJQj3gv80ss+F+KAM8n X7sEWJNX/Vv0s4IIss1fybiwD2xJZwTfdpGR+3hO3PJB+/lHxpNVOrsCuY9aqA7T6Zuo cH469X5zFwKsK+k7f7hfz8ijP9wO7g9LjG4jyXRuw1TnM4ccysND17fCJ05iCWTm0z19 dEgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gllm.fr header.s=fm3 header.b=cKCFI1hC; dkim=neutral (body hash did not verify) header.i=@messagingengine.com header.s=fm3 header.b=EI+eFq+B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gllm.fr Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id l9-v6si1075220wra.6.2018.07.03.08.06.04; Tue, 03 Jul 2018 08:06:09 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gllm.fr header.s=fm3 header.b=cKCFI1hC; dkim=neutral (body hash did not verify) header.i=@messagingengine.com header.s=fm3 header.b=EI+eFq+B; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gllm.fr Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0203F68A816; Tue, 3 Jul 2018 18:05:59 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1F809689812 for ; Tue, 3 Jul 2018 18:05:52 +0300 (EEST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 8701A21D28 for ; Tue, 3 Jul 2018 11:05:54 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Tue, 03 Jul 2018 11:05:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gllm.fr; h=date :from:message-id:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=ib6iP9f0hcB6CW4oNdjdYWRd4loCHpzEBYvQO8t9/Ok=; b=cKCFI 1hCvaIGEO1yUje5GU7LHvAVBETR6xUi0VKzIZNT3IC5hIFV/2mYIrmkHtCgoIV4m ghvqGcMxR58vgx6OLFvP1s7GzVh6OzzEUMGBi7fcO+1jK+u9NKy1cRFPM615tcf1 /8JlTxtPPA+5bOdmGNydKVCYtSCHB6nzXYkRQILfIIoefthEXDOlGHSFDBu7r7hW iP9tJ8sySUSFDYEjEsopFKcEmE8nUcvv/mOXji1FscbYYtdgwiOuQgmLmhHpU0MM /4Afx3wNnbMfHEwD1JcKEpKwPEvgpZ/YYa8Q5sRQKi3Dcimx2AkShZeZdIfLNrAZ yVDG08RDQsMttn0cw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=date:from:message-id:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=ib6iP9f0hcB6CW4oN djdYWRd4loCHpzEBYvQO8t9/Ok=; b=EI+eFq+BSZnR+bbBqE5mvqfb0ox8+vmAm hGpPtb1ive0gvvUYELf9lRkyDkyNmXveGAY7Zgy0lzTv5wfa7ckrDAIRb75zyfP+ YYwbvgt328o+RILlwvwC/5nMdmvWB2pdMaFeJKpmt8i28wShi/mCFCmWUV/nRhfx 8eCK1ZJFJjgidcmsndo9clCPovNjPWuYHiE8zQYquA52ZeEEyzi1Spzqarl1uud5 qu1lA1+sGRphZs9mWTWbvSxN3tl5myZuv/cWeuc+DEc81ztNg4wU8R17psxwNwBV tcYzJg+OgKSEd3OPm0DLYtp82uA2leWau/xhURcxOwh0742iImozg== X-ME-Proxy: X-ME-Sender: Received: from localhost.localdomain (unknown [78.194.250.245]) by mail.messagingengine.com (Postfix) with ESMTPA id B8601E4117 for ; Tue, 3 Jul 2018 11:05:53 -0400 (EDT) From: Thomas Guillem To: ffmpeg-devel@ffmpeg.org Date: Tue, 3 Jul 2018 17:05:50 +0200 Message-Id: <20180703150550.17177-1-thomas@gllm.fr> X-Mailer: git-send-email 2.18.0 Subject: [FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On macOS, a zero rc_max_rate cause an error from VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits). on iOS (depending on device/version), a zero rc_max_rate cause invalid arguments from the vtenc_output_callback after few frames and then a crash within the VideoToolbox library. --- libavcodec/videotoolboxenc.c | 72 ++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 35 deletions(-) diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index ac847358ab..050e5cefee 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -1019,44 +1019,46 @@ static int vtenc_create_encoder(AVCodecContext *avctx, if (vtctx->codec_id == AV_CODEC_ID_H264) { // kVTCompressionPropertyKey_DataRateLimits is not available for HEVC - bytes_per_second_value = max_rate >> 3; - bytes_per_second = CFNumberCreate(kCFAllocatorDefault, - kCFNumberSInt64Type, - &bytes_per_second_value); - if (!bytes_per_second) { - return AVERROR(ENOMEM); - } - one_second_value = 1; - one_second = CFNumberCreate(kCFAllocatorDefault, - kCFNumberSInt64Type, - &one_second_value); - if (!one_second) { - CFRelease(bytes_per_second); - return AVERROR(ENOMEM); - } - nums[0] = (void *)bytes_per_second; - nums[1] = (void *)one_second; - data_rate_limits = CFArrayCreate(kCFAllocatorDefault, - (const void **)nums, - 2, - &kCFTypeArrayCallBacks); - - if (!data_rate_limits) { + if (max_rate > 0) { + bytes_per_second_value = max_rate >> 3; + bytes_per_second = CFNumberCreate(kCFAllocatorDefault, + kCFNumberSInt64Type, + &bytes_per_second_value); + if (!bytes_per_second) { + return AVERROR(ENOMEM); + } + one_second_value = 1; + one_second = CFNumberCreate(kCFAllocatorDefault, + kCFNumberSInt64Type, + &one_second_value); + if (!one_second) { + CFRelease(bytes_per_second); + return AVERROR(ENOMEM); + } + nums[0] = (void *)bytes_per_second; + nums[1] = (void *)one_second; + data_rate_limits = CFArrayCreate(kCFAllocatorDefault, + (const void **)nums, + 2, + &kCFTypeArrayCallBacks); + + if (!data_rate_limits) { + CFRelease(bytes_per_second); + CFRelease(one_second); + return AVERROR(ENOMEM); + } + status = VTSessionSetProperty(vtctx->session, + kVTCompressionPropertyKey_DataRateLimits, + data_rate_limits); + CFRelease(bytes_per_second); CFRelease(one_second); - return AVERROR(ENOMEM); - } - status = VTSessionSetProperty(vtctx->session, - kVTCompressionPropertyKey_DataRateLimits, - data_rate_limits); + CFRelease(data_rate_limits); - CFRelease(bytes_per_second); - CFRelease(one_second); - CFRelease(data_rate_limits); - - if (status) { - av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate property: %d\n", status); - return AVERROR_EXTERNAL; + if (status) { + av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate property: %d\n", status); + return AVERROR_EXTERNAL; + } } if (profile_level) {