From patchwork Sun Dec 17 11:01:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhao Zhili X-Patchwork-Id: 45187 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:1225:b0:181:818d:5e7f with SMTP id v37csp6749589pzf; Sun, 17 Dec 2023 03:01:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IE9YoxSDobz4eCfV8eZ5RcRa7CqYDbTJOD31SVxvJPPWgzJU2A6EsyAVISItq6cQd6gA6cS X-Received: by 2002:a50:9f68:0:b0:553:4908:569f with SMTP id b95-20020a509f68000000b005534908569fmr34763edf.34.1702810904716; Sun, 17 Dec 2023 03:01:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702810904; cv=none; d=google.com; s=arc-20160816; b=GPB3WO0Fznef3+sWNaxyUS+8VitybEcx9Rsh5p4Vw1QHeZVyUj/6Xh7I3K4Yvi15hX bRFTAcOcmwpR7iLvOL9NO3BosroJu7engfZ1qumu6/lqD1/iABTLiO2LNoNs4a1wpJee GeE8zEWneWxPvJZzc5YZinnowwO3cp9rxXeu3jN3THzTK6wuaCJYYOOtisnwDH34otuf 4w19oUDbE1fuVrBHhuEoXWrof74+FZw5VZ8jLE20r8x5ALiwn7gx2aVlTpPama/uUf7l jV4JcZzqTD+CzJ/GDUihQlgcXP0SIC+nDFfrVegLOfGBphai1Usb7Wbkfh9oh33Fqia+ teGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:date:to:from:message-id :dkim-signature:delivered-to; bh=qsFnFcbluztpyFi9V6iMroiOnbQB5FmH4IIiweFFc7M=; fh=HnHYuZ9XgUo86ZRXTLWWmQxhslYEI9B9taZ5X1DLFfc=; b=yaobe7bUCxo87eg66DcQ3SesaBh+4MMDnPtX2EiP6Ca0GUWULn0Gj0kw3SbVjf88lY BCtn4zrOjj88yi+RISmh+2+rIEq60u84HS5D2/W6AywrejKBEjpbIUZDG7Zu5kdwUV3N Ctc7N/iHuhxbiSt7ikZxbRiU30DZCiyGipiM2cMDt27RFQrS7mkiRrOCxClfe3HeyjD+ mgOpZUGL1OQWRNaVq3g+xbZtdQX2ozJKsLjV2u+tY5OtmOvSVLZE+TNdtCI/WBQKeaIO 4M6tK64BSDfoMlZ0XkfqiZlswE64CdxGxo98f6jdmraqqyneNgOijOLOeTi/LknzIe6K SIrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="LOI/Kx6r"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id s26-20020a05640217da00b00552811b12dcsi2944282edy.254.2023.12.17.03.01.44; Sun, 17 Dec 2023 03:01:44 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@foxmail.com header.s=s201512 header.b="LOI/Kx6r"; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=foxmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id E131468D147; Sun, 17 Dec 2023 13:01:29 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from out203-205-251-72.mail.qq.com (out203-205-251-72.mail.qq.com [203.205.251.72]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6377068D035 for ; Sun, 17 Dec 2023 13:01:20 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1702810872; bh=hHdjoQgcb3LsuqKPCrkSwPEvOikACzVJob3ZBWHP16I=; h=From:To:Cc:Subject:Date; b=LOI/Kx6r6u4gWUqKInFRvclHvvHv4JNBT06sUq5jpKvbnYbSiXJqIx5XAoBXIBIIr v9Hs9kLBB6dx2w3VooPnFgNc5He0Effmb5T7e/AzkPpVvnJwq4MPFINJWOfbv+RcQr oUZqHA6Tron2cqmYx0TX0IPSXxTq/Et9JZpSUhwQ= Received: from localhost.localdomain ([113.118.106.67]) by newxmesmtplogicsvrszb6-0.qq.com (NewEsmtp) with SMTP id 4B23E98; Sun, 17 Dec 2023 19:01:11 +0800 X-QQ-mid: xmsmtpt1702810871tgbuap8pb Message-ID: X-QQ-XMAILINFO: MIAHdi1iQo+z3SLlzHZjQQMDBMMeVjQFno5cZ0yohL+lAgnRhlBd9vQzr/ytHY nEwkJLqTKr6v8JHMTmax/P2uNXt1brDsElc/zAtwT24xcJV0xda0qa/4LjuS8o1SoggfmVAU0NEf wJFYHk496SLIUqlSEXp+HZcVdqW0N8q159a4gGA6n6wweH3Mi4zj2lxBCqIeUXVNEvzopB6EJCSj PpwUkL3v8Vz7kuO7L76dNLELXbxxXYQdO2Yqv170Nlym+dAjUgpOCSLDhiAZt8T6arK2cly9D6TC rogordvTqOdVaXk0ogMPsHerFbyPtG8b5+WlQrYeQyVk+wYjGvHamGmIGh/dEb7u9CxfPE2TzxPe GWO2ppT9vOhC5frh9wfCGfsjdKihthn/YRf1iJf+p8E/4S6+myZlS06/pZBZHIn0idEJ7dWn5ZP0 zUOrXEOMq3qYZtIh7qqgfZlEpOQHQGQlpacfYWmzlx5aR3P5Xg5h9wbxz8UWBwPBBqynD26Oz6O7 Q0Rc/QDRlXes4PB3szbrvJ+FIBdwd982PGTdwjdv9crbJd8Yos6xH3tT5cBlna4i1d+tHCLeCBud 2uBp9mgX0EL1KHMI2Rf9du59K8lyaFfaO4AvKoqIcI2Dv0UzWGkeRRyZKhJodax7IB5Nvo0ftG5B 7nPqjq8WPIfPcLEvDnR+MqT4+zF0q2+Qvvl1beImC4ZhREejNVKly2dSwWdUUbhLfZrf53b+Ixeg l0NZg3B94ofJnrKNWvDMJtENLryf+dJHe6R6JEIHoQLXUByDMy3t0SZjqieSOl+L2n1rGP9xy7e9 cmEHmJ9Fmg/Qna5dgwtri36sMDddIrGD6WM+eh8cCOEbmJ6ZZULl+06KWULDHHf3TqyokGM1Umud Gi+MxdfeSebktRBZid+B6pCmSLSV02ZuaYYrRz91xYqLKvK1ARJ4lObaan1CjwHlyo5qeG92aDmO h+wXMY1Gsalp2izBpnxQKTTaWIuNZ5FExrmjBSbTw= X-QQ-XMRINFO: Nq+8W0+stu50PRdwbJxPCL0= From: Zhao Zhili To: ffmpeg-devel@ffmpeg.org Date: Sun, 17 Dec 2023 19:01:08 +0800 X-OQ-MSGID: <20231217110109.264409-1-quinkblack@foxmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 1/2] fftools/ffmpeg_filter: fix null pointer dereference X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Zhao Zhili Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: teaL78nkB86j From: Zhao Zhili A dummy frame is created with format NONE passed to enc_open(), which doesn't prepare for it. The null pointer dereference happened at av_pix_fmt_desc_get(enc_ctx->pix_fmt)->comp[0].depth. frame #0: 0x0000005555bc34a4 ffmpeg_g`enc_open(opaque=0xb400007efe2db690, frame=0xb400007efe2d9f70) at ffmpeg_enc.c:235:44 frame #1: 0x0000005555bef250 ffmpeg_g`enc_open(sch=0xb400007dde2d4090, enc=0xb400007e4e2daad0, frame=0xb400007efe2d9f70) at ffmpeg_sched.c:1462:11 frame #2: 0x0000005555bee094 ffmpeg_g`send_to_enc(sch=0xb400007dde2d4090, enc=0xb400007e4e2daad0, frame=0xb400007efe2d9f70) at ffmpeg_sched.c:1571:19 frame #3: 0x0000005555bee01c ffmpeg_g`sch_filter_send(sch=0xb400007dde2d4090, fg_idx=0, out_idx=0, frame=0xb400007efe2d9f70) at ffmpeg_sched.c:2154:12 frame #4: 0x0000005555bcf124 ffmpeg_g`close_output(ofp=0xb400007e4e2d85b0, fgt=0x0000007d1790eb08) at ffmpeg_filter.c:2225:15 frame #5: 0x0000005555bcb000 ffmpeg_g`fg_output_frame(ofp=0xb400007e4e2d85b0, fgt=0x0000007d1790eb08, frame=0x0000000000000000) at ffmpeg_filter.c:2317:16 frame #6: 0x0000005555bc7e48 ffmpeg_g`filter_thread(arg=0xb400007eae2ce7a0) at ffmpeg_filter.c:2836:15 frame #7: 0x0000005555bee568 ffmpeg_g`task_wrapper(arg=0xb400007d8e2db478) at ffmpeg_sched.c:2200:21 Signed-off-by: Zhao Zhili --- This bug is trigged by 10bit H.264 which doesn't supported by mediacodec. There is some misleading error messages from ffmpeg cli: [h264_mediacodec @ 0xb400007513788010] Failed to dequeue output buffer (status=-10000) [vist#0:0/h264 @ 0xb4000074a3786b30] Error submitting packet to decoder: Generic error in an external library [h264_mediacodec @ 0xb400007513788010] Failed to dequeue output buffer (status=-10000) [vist#0:0/h264 @ 0xb4000074a3786b30] Error submitting packet to decoder: Generic error in an external library [vist#0:0/h264 @ 0xb4000074a3786b30] A decoder returned an unexpected error code. This is a bug, please report it. [vist#0:0/h264 @ 0xb4000074a3786b30] Error processing packet in decoder: Internal bug, should not have happened fftools/ffmpeg_filter.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index bb755d7bb4..9dc47f9d90 100644 --- a/fftools/ffmpeg_filter.c +++ b/fftools/ffmpeg_filter.c @@ -2192,7 +2192,7 @@ static int close_output(OutputFilterPriv *ofp, FilterGraphThread *fgt) // we are finished and no frames were ever seen at this output, // at least initialize the encoder with a dummy frame - if (!fgt->got_frame) { + if (!fgt->got_frame && ofp->format != -1) { AVFrame *frame = fgt->frame; FrameData *fd;