From patchwork Mon Jan 1 16:28:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derek Buitenhuis X-Patchwork-Id: 7062 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.79.195 with SMTP id r64csp13752296jad; Mon, 1 Jan 2018 08:29:33 -0800 (PST) X-Google-Smtp-Source: ACJfBotIsCc1lz9oD8Fc2O1PLnBDPgFsQrzCBa1790iMqKZ7OSb+Vc1v3Y6NMey98Y4C0uEVgutU X-Received: by 10.28.49.195 with SMTP id x186mr31940308wmx.116.1514824173838; Mon, 01 Jan 2018 08:29:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514824173; cv=none; d=google.com; s=arc-20160816; b=0BwLj8DZ8c6cxC0LJdoQEy+jaAp59a52YPPA9e/wYTsoQBH1kwm0R0h9b0os0obSn7 grII4hAmJQpk9PDr1LCTiNxGK+37BdK7rq/eu6FJZwmyDE8t4XzfP26mY+eQaj8saXQg Fz46eXM8Zeof+7GcCtJijyhSy6rtt1ortAMkp/c6IDsj+SBMw7xv+VjCvKCPizi1NxkX khvF9FA2gnkkhw5WeIDBD92uknZwUe/5NBQXtOGrydDwViQwazdprQxYbKtUiitLsysa 7Rw/6HjNzbkjm9V7xv/eQmCRkMVcnimxhCo48mM4nuSJC3UnJHzGoZTXiFw5m8uZy5Dx zFsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to:arc-authentication-results; bh=EOWVmkJ1beFm3ItmbnVHuHU8RkthN87cLntjXYnUuLw=; b=qeO/loxzPTgu3uWTg7d4SBzjDSor/aGUTlsyR73QuXQ6oV8GhoariAguG7SVaDwmGg dD7uTyHXruImWvc939HXF1lLVxtwKU+BELSpVX/eBKUOraCTUB9zPQhjWwmz3L1MtFjH CMd78lvSh3Qrj53B8JhajikAsZhJUw3RKMgHsPhCtSaskM/h40FlmxO5Xvzq+Rx6oAr9 +TldD+AliSpBLy9+vD3bJkWj46YZFR1pOSHzwIzaFCqdx6dacJsKb6Y2Ipq9mV/9rTLB 3jF7e0LM3Q4WDILkxi2CHyNB+HH7poH1v1EW3IVsE96PiI9eNCl8cUTJyxOBHo8h0bKV QYuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QPG9KTID; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id w10si33158640wrg.453.2018.01.01.08.29.33; Mon, 01 Jan 2018 08:29:33 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QPG9KTID; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 74195687EF1; Mon, 1 Jan 2018 18:29:09 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2218C680375 for ; Mon, 1 Jan 2018 18:29:03 +0200 (EET) Received: by mail-wr0-f195.google.com with SMTP id p69so37850481wrb.8 for ; Mon, 01 Jan 2018 08:29:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=6CWz5oWKV/1EjsuXp1ZlPTSkkt26Q0TJaSVUaUkn8+g=; b=QPG9KTIDiJHRKWs2kQLPS5ZZqsWe1AN2VjntVgoTrlzIhLY4BEMzfH4neNeWfN3gJW dqCzymvXB2gwAl9KwM8IZjM954aTRUWWMNikK6QBe4TK8P6fC4cB/cehUgc79FYkVfJM JI5rmp0A5XzQXErtP9LrItnCwpheQiu1dc+WvINmAd3wDxuIo3vhkKhXjS6TK4nCleVN ME9DCuH408M8p5/Mhw1pNryE0xXjCIv4MqTqykgux148ACDCHPK0ayCLI2dc7/eEksA/ szzwrLbQxU2t1J5Loy+3oEVbps8pwlb7SmRLCavewDkUYiqVH4wE9a6xmfjk09G8XGx6 ryUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=6CWz5oWKV/1EjsuXp1ZlPTSkkt26Q0TJaSVUaUkn8+g=; b=Nt+GivoXvxdBNym0jEmtQSvPbVQREQKxoWAb7eQUpPrzShovfbu0V4rxFbfMz5Adyy OgYW0wT8772IK1+BcjX88jN5zfj/fyXXWP8HCMMdaA8GB8uvJalCBGmV2VMxUe6lK7ZU eMbs/pakgFEIztDgKcssCnAiZMAc+nmJL1ZOJdU38F7Y02Md3smvhGX8zYVwEpaXvnBG FZH7V23QCJMu9NP5PFzctELKJ0rVGPDwBDcPCZF6/efeeUY+oe29Q1wu0FPP8wpSvEXY Wkb7mJKXtYUw/ff06kTsywsL7/yJ9FFsAsjejUrg20fSV97BkxIj/vbnb2vOLYJ1tQOO Llpg== X-Gm-Message-State: AKGB3mIIwJLFOj+FCf8nr4oe2b4wKLsFtrfAr4RDqJQtOoBP73OKzEUI BAdLe7VLLu/fRDRc1Q5GM2V8E+kP X-Received: by 10.223.181.135 with SMTP id c7mr13503745wre.272.1514824158346; Mon, 01 Jan 2018 08:29:18 -0800 (PST) Received: from vimeo-vm.localdomain ([154.57.238.96]) by smtp.gmail.com with ESMTPSA id o16sm34576961wmi.29.2018.01.01.08.29.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 01 Jan 2018 08:29:17 -0800 (PST) From: Derek Buitenhuis To: ffmpeg-devel@ffmpeg.org Date: Mon, 1 Jan 2018 11:28:36 -0500 Message-Id: <1514824117-15587-2-git-send-email-derek.buitenhuis@gmail.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1514824117-15587-1-git-send-email-derek.buitenhuis@gmail.com> References: <1514824117-15587-1-git-send-email-derek.buitenhuis@gmail.com> Subject: [FFmpeg-devel] [PATCH 1/2] vf_paletteuse: Add error checking to apply_palette X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" This fixes a segfault caused by passing NULL to ff_filter_frame when an error occurs. Signed-off-by: Derek Buitenhuis --- libavfilter/vf_paletteuse.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/libavfilter/vf_paletteuse.c b/libavfilter/vf_paletteuse.c index 1980907..ede2e2e 100644 --- a/libavfilter/vf_paletteuse.c +++ b/libavfilter/vf_paletteuse.c @@ -894,9 +894,9 @@ static void set_processing_window(enum diff_mode diff_mode, *hp = height; } -static AVFrame *apply_palette(AVFilterLink *inlink, AVFrame *in) +static int apply_palette(AVFilterLink *inlink, AVFrame *in, AVFrame **outf) { - int x, y, w, h; + int x, y, w, h, ret; AVFilterContext *ctx = inlink->dst; PaletteUseContext *s = ctx->priv; AVFilterLink *outlink = inlink->dst->outputs[0]; @@ -904,7 +904,8 @@ static AVFrame *apply_palette(AVFilterLink *inlink, AVFrame *in) AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); if (!out) { av_frame_free(&in); - return NULL; + *outf = NULL; + return AVERROR(EINVAL); } av_frame_copy_props(out, in); @@ -918,21 +919,25 @@ static AVFrame *apply_palette(AVFilterLink *inlink, AVFrame *in) av_frame_make_writable(s->last_in) < 0) { av_frame_free(&in); av_frame_free(&out); - return NULL; + *outf = NULL; + return AVERROR(EINVAL); } ff_dlog(ctx, "%dx%d rect: (%d;%d) -> (%d,%d) [area:%dx%d]\n", w, h, x, y, x+w, y+h, in->width, in->height); - if (s->set_frame(s, out, in, x, y, w, h) < 0) { + ret = s->set_frame(s, out, in, x, y, w, h); + if (ret < 0) { av_frame_free(&out); - return NULL; + *outf = NULL; + return ret; } memcpy(out->data[1], s->palette, AVPALETTE_SIZE); if (s->calc_mean_err) debug_mean_error(s, in, out, inlink->frame_count_out); av_frame_free(&in); - return out; + *outf = out; + return 0; } static int config_output(AVFilterLink *outlink) @@ -1011,7 +1016,7 @@ static int load_apply_palette(FFFrameSync *fs) AVFilterContext *ctx = fs->parent; AVFilterLink *inlink = ctx->inputs[0]; PaletteUseContext *s = ctx->priv; - AVFrame *master, *second, *out; + AVFrame *master, *second, *out = NULL; int ret; // writable for error diffusal dithering @@ -1025,7 +1030,9 @@ static int load_apply_palette(FFFrameSync *fs) if (!s->palette_loaded) { load_palette(s, second); } - out = apply_palette(inlink, master); + ret = apply_palette(inlink, master, &out); + if (ret < 0) + goto error; return ff_filter_frame(ctx->outputs[0], out); error: