From patchwork Sun Oct 11 00:09:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 22848 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 30D7B44B600 for ; Sun, 11 Oct 2020 03:09:33 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 0229E68BAA3; Sun, 11 Oct 2020 03:09:33 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id C381468BA51 for ; Sun, 11 Oct 2020 03:09:26 +0300 (EEST) Received: by mail-wr1-f65.google.com with SMTP id s9so2493355wro.8 for ; Sat, 10 Oct 2020 17:09:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=alZvlGIglMJkxPAf1MImeulsEDxej8DbgcgqlXjPh+0=; b=pZmC75iC1g5PhDsN47qg5xVlPeQS10w182bxpxCeRwTR6bbjsy5j1/pDwffm/W3P3H 3+SCL+kur18D/H0QGsSkpj3T68nqTxiV/tk1aKMyZgDP9Fvysvjsna5kqJm8SiPYsxXR LYQ8QleTl6D9GvA/yZ1/Xg3yWgtOqFBosQR7NB52Ad+Q0y/U0fMVQDMtE8bb+iAGMFbD Acq1VTc8CHLErYkqhVHLxo5VOWwMfYm126jUW5CNWTcTnuIGriFzyNw5E8kfFculztrI x/l8skZuWCgfPCypdu8IhOIf8V4SwUh3vkihzZqGPMqQN4wSsIU//dl3Cw4SiIkEQZfS PC0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=alZvlGIglMJkxPAf1MImeulsEDxej8DbgcgqlXjPh+0=; b=LovmWASJwHyXjognFb6IFniLlvMM2drhdbEVIm1fE//0vtldGO243Yl5d+Sa2bhaZs THcZ41hfd35hdQrClJZcVEd+MdWw/RmRRjUQJX5ydq3vYvQ1ct4uHq6FDXx8tTbmiBuZ 0mUQkR6NAesiaQiJkTuxrQHmYeQO2xJFwQapoflIg8NG++/144MbFQLX8HPBP0dLC0Xf Gt5vrMXdFMcYF9JRUmPDRxXfamIxm5jQV+QqbnTK7pVl0PwBJgv7tHvrKEl3NfPuKgIl akyAbalXHdIyQ8yNvhwu35tyuJbVvEu35QZQ0uWurAo/n/EKWVaCMwGwTptO55Ny1mvM qNMQ== X-Gm-Message-State: AOAM533bVU+LHqJsc6A7COnE2+FT9v9me5tS7czkDQ/OP4nslN2C0TmW VCBILn0LIFkVOGfVBM7ucQuFaXTbR9Q= X-Google-Smtp-Source: ABdhPJxvFziEInel3bUnm1lpgTYyogaRQp51QoojQPvoCURujegXt3SEy/SVAILwwaXMyqtQQ4Gk6Q== X-Received: by 2002:adf:ec06:: with SMTP id x6mr21879957wrn.404.1602374966035; Sat, 10 Oct 2020 17:09:26 -0700 (PDT) Received: from sblaptop.fritz.box (ipbcc1aa4b.dynamic.kabel-deutschland.de. [188.193.170.75]) by smtp.gmail.com with ESMTPSA id e15sm17774529wro.13.2020.10.10.17.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Oct 2020 17:09:25 -0700 (PDT) From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Sun, 11 Oct 2020 02:09:18 +0200 Message-Id: <20201011000918.1137879-1-andreas.rheinhardt@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/vp9: Fix stack-buffer overflow with VP9 VDPAU available X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" ccca62ef991f0a47dfa30c3e822d91294b8afe4c added new VP9 VDPAU profiles and as a consequence AV_PIX_FMT_VDPAU can now be twice in the list of pixel formats used for format negotiation by ff_thread_get_format(); yet there is only one entry in said list reserved for VDPAU, leading to a stack-buffer overflow. This commit fixes this by making sure that AV_PIX_FMT_VDPAU will not occur twice in said list. Fixes Coverity ticket 1468046. Signed-off-by: Andreas Rheinhardt --- Other solutions would be a) Counting CONFIG_VP9_VDPAU_HWACCEL twice for the size of the pix_fmts array (as is already done for D3D11VA). But this would leave AV_PIX_FMT_VDPAU twice in the list which seems wrong. b) Adding a break inside the #if CONFIG_VP9_VDPAU_HWACCEL block for YUV420P. But then the other hardware accelerations won't ever get a chance. c) Let AV_PIX_FMT_YUV420P directly fall through to AV_PIX_FMT_YUV420P10 without adding a pixel format there at all; AV_PIX_FMT_VDPAU would then be added at the end, changing the current priorities, unless it would be moved to the beginning of the AV_PIX_FMT_YUV420P10 block. libavcodec/vp9.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c index 8b89fd68e2..d04e9fc062 100644 --- a/libavcodec/vp9.c +++ b/libavcodec/vp9.c @@ -225,7 +225,8 @@ static int update_size(AVCodecContext *avctx, int w, int h) *fmtp++ = AV_PIX_FMT_VAAPI; #endif #if CONFIG_VP9_VDPAU_HWACCEL - *fmtp++ = AV_PIX_FMT_VDPAU; + if (s->pix_fmt != AV_PIX_FMT_YUV420P) + *fmtp++ = AV_PIX_FMT_VDPAU; #endif break; case AV_PIX_FMT_YUV420P12: