From patchwork Wed Jul 4 07:05:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Guillem X-Patchwork-Id: 9604 Delivered-To: ffmpegpatchwork@gmail.com Received: by 2002:a02:104:0:0:0:0:0 with SMTP id c4-v6csp461759jad; Wed, 4 Jul 2018 00:05:36 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeadt5tU2nkYrZOZU+3qWHTFJq7Zx9h5RcWisGGXWqaWq/oQWATFg81f97kkoH8J+ApJC2j X-Received: by 2002:a1c:d681:: with SMTP id n123-v6mr692792wmg.158.1530687936456; Wed, 04 Jul 2018 00:05:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530687936; cv=none; d=google.com; s=arc-20160816; b=0JJtsvrB5zTWqn3Fuq91nUlfwVh6zHGVqtTrVep/AufkQswCzM7g6K0FBnAu8w9h3/ qIgZTrGWlcdB51BTqICo0AjR5/pf3vSD6jvGNSOcjQfeHX2HBIt0vqrF44EopLHm8R5j VRDMotHOr7yXlI2t7Q4Q0raxMcwxRSyiX3Gfvu5TO5G2fd9pphIDBkavtunzEYX8E9rQ zrxeqCkfstJ+tBVSAxgAsL3nuhZCA5ZUaQum9Z9gp0loafMhX8XO3kNScRMVQP1N9zXB MYYeqTFafKzTaOdgxzYPuKB6nqQQsyvMDSHRkWMquaB3uJ4fCcseDS3s3r5sft3p4yAV HoqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :dkim-signature:delivered-to:arc-authentication-results; bh=w/bWxCva1s79T1dk12Hp0EahO0fLk0MKLA1ZRwq2duw=; b=cjaozCSipZ69aLr75+M887uvLdTLM5dSDArUvJwxXZS7xK+Qqlt/sEFAKlVIVoqeYI viz7gXl724QrTL9smQRTVpULxBwDhSB2eUuWffmjTL07yu+l6YLNsh8o3o9h+upF8eob KXjccsn6WsPWvulmyHDCd19eiWrOvK4kNro/5Hj73UOwj+/LbQgfjo2FzXmSvuypOD2t ag42jldSZ9j3p5lxqLl7lFU2MhXtfmooG7iwbeGDZSFw/dNWoyN6vP4CjuJumMQNK+RL oi/XlBSdV8lvlRPB5+m0vBVWKvHuBNyNuy9LSmmZvm50sS1S34Oqwu96lf1gkA+bth2H Bb2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gllm.fr header.s=fm3 header.b=cKx7H33E; dkim=neutral (body hash did not verify) header.i=@messagingengine.com header.s=fm3 header.b=WQ8nV4aF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gllm.fr Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id b1-v6si2162145wro.113.2018.07.04.00.05.36; Wed, 04 Jul 2018 00:05:36 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gllm.fr header.s=fm3 header.b=cKx7H33E; dkim=neutral (body hash did not verify) header.i=@messagingengine.com header.s=fm3 header.b=WQ8nV4aF; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gllm.fr Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 653CF68AA34; Wed, 4 Jul 2018 10:05:30 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A503268AA33 for ; Wed, 4 Jul 2018 10:05:23 +0300 (EEST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 316AE21F3C for ; Wed, 4 Jul 2018 03:05:27 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 04 Jul 2018 03:05:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gllm.fr; h=date :from:message-id:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=U/X7tvRZqAENPv/jfeOTgO2BDbGN4LecyGeUV4Q1F0w=; b=cKx7H 33Ex6JB9ZWPLc0Goqb3LQ3+Fla94r1shCvrdEeigFwXgYbWVT0z1Nj9whVUE+jHl kzmPPP/QXXlTbGj7DCiJmYHly8ftn6vBLPpV0n5qxZ+3/BNCKR8lv7Wea8+/jFKx z3ff6w4P7wsd4YxlbxY5pv1WS6x58vsBu6rLtpdP9TZgyy1ikxc++9vDZoK0B7N8 R0Jn2NSAjUribxW7h+/2IuTLICUqEltbtlhxUjJzFJuHVPd8deimLOqvHogPlewy prWGqsYhKkbcE63XIiQPpVO5iXgfMLAQGJWLgfADGan1l8rxCGrEVh7Z434J5mOG YS5T5eTdLy9FgTZMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=date:from:message-id:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=U/X7tvRZqAENPv/jf eOTgO2BDbGN4LecyGeUV4Q1F0w=; b=WQ8nV4aFvyKqzFQ4IukVqeA5yGYjyirrP LliX1QeABkvbnL3doiH6/LzCZnLh0zR9vRcA5N8zk+WnxKR99GP5Uc8pVDk05jhZ ECIkGSUoM1T1bXt6sPlSEuzfCEG1ug8uO88641+muOklegdzZD7q8bsH3Y/q150L hKSlIiYKcvPHAv88dywi7AX/MwjPUiOlEJjiXtRXdbUwMglC+ZJa97mP0r80GRpO BQT2VQf3/Ja6pV/a8N58q9NM6L7+0e7MegYYxBZZ/SIRHHcDnnTCIROJCW6nQisa OTfxf2F2H3+M/C5jiXTMALs5v3Q8O0zfD0ogzX3y/F3lU928ayOXw== X-ME-Proxy: X-ME-Sender: Received: from localhost.localdomain (unknown [78.194.250.245]) by mail.messagingengine.com (Postfix) with ESMTPA id 86D86E463E for ; Wed, 4 Jul 2018 03:05:26 -0400 (EDT) From: Thomas Guillem To: ffmpeg-devel@ffmpeg.org Date: Wed, 4 Jul 2018 09:05:22 +0200 Message-Id: <20180704070523.2573-1-thomas@gllm.fr> X-Mailer: git-send-email 2.18.0 Subject: [FFmpeg-devel] [PATCH 1/2] avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" On macOS, a zero rc_max_rate cause an error from VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits). on iOS (depending on device/version), a zero rc_max_rate cause invalid arguments from the vtenc_output_callback after few frames and then a crash within the VideoToolbox library. --- libavcodec/videotoolboxenc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index ac847358ab..aa9aae7e05 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -1019,6 +1019,7 @@ static int vtenc_create_encoder(AVCodecContext *avctx, if (vtctx->codec_id == AV_CODEC_ID_H264) { // kVTCompressionPropertyKey_DataRateLimits is not available for HEVC + if (max_rate > 0) { bytes_per_second_value = max_rate >> 3; bytes_per_second = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt64Type, @@ -1058,6 +1059,7 @@ static int vtenc_create_encoder(AVCodecContext *avctx, av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate property: %d\n", status); return AVERROR_EXTERNAL; } + } if (profile_level) { status = VTSessionSetProperty(vtctx->session,