From patchwork Fri Dec 25 15:47:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 24649 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 357E544B6ED for ; Fri, 25 Dec 2020 17:53:05 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 018D768AE63; Fri, 25 Dec 2020 17:53:05 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 453B968AA85 for ; Fri, 25 Dec 2020 17:52:58 +0200 (EET) Received: by mail-ed1-f53.google.com with SMTP id r5so4355356eda.12 for ; Fri, 25 Dec 2020 07:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=XGwW8OLPuFMM76bgFaXS/6HXtxInUziDe2TO4WwUC3s=; b=I6WJAspIZzekcs1Nkl2M/3sAsSz2FeKpAlcc3w6R2EsK2H8c4u0qrE4tNiSS1WoDYl F8JxY5taGkNuFYxr+tO141mhFiF8QhOPqHSjoqJboGMl3Sa68h+60uYeYq8XFs/kNSmm w07Hieg43EfM0Vb7UEQ+AgHfa8rG9/CXvC2m1KpQwJuWQgUQMAPcIrVkiprQsU+hLjQs yq9//q5iNqGjBKE2fbjesh2/2dE8YIk4V+k2qAriZEL4+lZi9YNIomjYlkxT7oofec8l yZSWHHD6fFImp0znlng5kLVRnRfEMezZW0l0EcrlX2W3aleuf5m3sSQ4JA9YD11lqy3c g+ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=XGwW8OLPuFMM76bgFaXS/6HXtxInUziDe2TO4WwUC3s=; b=ctxpkNTguc7lIeVjFu4kLaW0mX3Hrc4uBOEbesbevQAWmoYSKOQ21fsFKvWKkrNB/4 Vcn20gnExOIXPd/69oea2Un2UYvj2YmdqksEfR3XTjPiEGc9kRMWC60yzSptrrnK3pLn 4NVaCMe9idy0F98pQZmMjSJa51kJh/qJRqbppbXrP21/sre+6PmNOQTaGH0imBcCYh3Z /kJHy/22HJj+CljINpjLMpRmvIykxy7Leu+m5+2dW077kdRR+Ri7qzNjqAq7/D7nRZ2N Uw9MGutcw0Ysq8UeBDhM3pkrXwCrhawoZXaYTt+jiSNfE0x3F9wQChLoJapo6knWhzBS N4Rg== X-Gm-Message-State: AOAM532IY7mEMdlRiNrXP7VadDibxf6wmMIkht1i/7CgSc1KoFS3GxTg 7kxfpDLXuydyvi43SKR5/lijnPNSKpw= X-Google-Smtp-Source: ABdhPJwF/7+FEtvdvVfEn/4nL0lrjQPBMOGEs+G+iITXJyFtjjD1f+bTKLIIF825iskDkITcCf2qGA== X-Received: by 2002:a05:6402:1041:: with SMTP id e1mr33723457edu.54.1608911577151; Fri, 25 Dec 2020 07:52:57 -0800 (PST) Received: from sblaptop.fritz.box (ipbcc1aa4b.dynamic.kabel-deutschland.de. [188.193.170.75]) by smtp.gmail.com with ESMTPSA id ho12sm13733010ejc.45.2020.12.25.07.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Dec 2020 07:52:56 -0800 (PST) From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Fri, 25 Dec 2020 16:47:19 +0100 Message-Id: <20201225154724.287465-2-andreas.rheinhardt@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201225154724.287465-1-andreas.rheinhardt@gmail.com> References: <20201225154724.287465-1-andreas.rheinhardt@gmail.com> MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH 2/7] avcodec/vc1dec: Postpone allocating sprite frame to avoid segfault X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Up until now, the VC-1 decoders allocated an AVFrame for usage with sprites during vc1_decode_init(); yet said AVFrame can be freed if (re)initializing the context (which happens ordinarily during decoding) fails. The AVFrame does not get allocated again lateron in this case, leading to segfaults. Fix this by moving the allocation of said frame immediately before it is used (this also means that said frame won't be allocated at all any more in case of a regular (i.e. non-image) stream). Signed-off-by: Andreas Rheinhardt --- libavcodec/vc1dec.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/libavcodec/vc1dec.c b/libavcodec/vc1dec.c index 7809234ff7..5cdf197da7 100644 --- a/libavcodec/vc1dec.c +++ b/libavcodec/vc1dec.c @@ -539,12 +539,6 @@ static av_cold int vc1_decode_init(AVCodecContext *avctx) ff_h264chroma_init(&v->h264chroma, 8); ff_qpeldsp_init(&s->qdsp); - // Must happen after calling ff_vc1_decode_end - // to avoid de-allocating the sprite_output_frame - v->sprite_output_frame = av_frame_alloc(); - if (!v->sprite_output_frame) - return AVERROR(ENOMEM); - avctx->has_b_frames = !!avctx->max_b_frames; if (v->color_prim == 1 || v->color_prim == 5 || v->color_prim == 6) @@ -577,20 +571,15 @@ static av_cold int vc1_decode_init(AVCodecContext *avctx) v->sprite_height > 1 << 14 || v->output_width > 1 << 14 || v->output_height > 1 << 14) { - ret = AVERROR_INVALIDDATA; - goto error; + return AVERROR_INVALIDDATA; } if ((v->sprite_width&1) || (v->sprite_height&1)) { avpriv_request_sample(avctx, "odd sprites support"); - ret = AVERROR_PATCHWELCOME; - goto error; + return AVERROR_PATCHWELCOME; } } return 0; -error: - av_frame_free(&v->sprite_output_frame); - return ret; } /** Close a VC1/WMV3 decoder @@ -1147,6 +1136,11 @@ image: avctx->height = avctx->coded_height = v->output_height; if (avctx->skip_frame >= AVDISCARD_NONREF) goto end; + if (!v->sprite_output_frame && + !(v->sprite_output_frame = av_frame_alloc())) { + ret = AVERROR(ENOMEM); + goto err; + } #if CONFIG_WMV3IMAGE_DECODER || CONFIG_VC1IMAGE_DECODER if ((ret = vc1_decode_sprites(v, &s->gb)) < 0) goto err;