From patchwork Wed Sep 25 15:20:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 15286 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id 381FA449159 for ; Wed, 25 Sep 2019 18:26:46 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 167A6689E13; Wed, 25 Sep 2019 18:26:46 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 99EA4688183 for ; Wed, 25 Sep 2019 18:26:39 +0300 (EEST) Received: by mail-wm1-f66.google.com with SMTP id f22so5456375wmc.2 for ; Wed, 25 Sep 2019 08:26:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=jWENyvwxwZ+ZZYvdBJWr11gbhJn8rQbJL5JgAJvbIrM=; b=e3T3gWQlxqTYw/2XYK1GPPUeu9a33ViTl3Cigy+0FscI3wbq6uZLmTJrblKCSF7L9B DxvbZzohIF6I4VQCcGNLxXcY+VuwQywqWH31fnNaw3MpT9GAi5Inpshd7KtLq7ZBOIA6 J3kygV/nxol1OKHX3hchAlZuGpWk/CIIMaANBZEa5UVry+mlmQ1p7cF7/Pxq8OmSNmOv M8jXMFk0BT12t+dMcyFBrvzKhRoNgKUNZlr7dkJK9rorIMLsaoinE35xuntfN/RqvPCG NHNPXCBhKe+SdCzupDGpVsKJa5CeR+IdyhCCv589q/QG7/USNsXAzdqOADdGX865POEY BwVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=jWENyvwxwZ+ZZYvdBJWr11gbhJn8rQbJL5JgAJvbIrM=; b=k6covLHSwNnC+3N0FEf4I8qMJMyQAUqoMjw1Tf5/sr2mNpGciyHTLaQpZGWiQWbkGf qMUeVHCkWTRZtxfghvcPwKP2L1eB644KQxf/2/yydMde862Xkd7tfEpbLIFLHgKONgRn s4FH5XexfQOF9nSDy2IYenRiP5UoUmG/gtVfQWu/4OzIqqKt+x8ogBiU3xL7OfCIYySy yh3nrCf27XG2OnYmdGMaBFdRTuxpkl0Z9snBd6wVownkHJALzCvjBnZiIylS3ajwneuq LrFhjkm4eqf9r8HU9E4j2Si7y98HYy4g3Zfg17oUMPU0N97jsz8kLmbKr3m0ePL9JZ4I FOAw== X-Gm-Message-State: APjAAAUV0uCPZ6Fl7zX0sISPYcXgMdVVt+Mpp1s8ol73oUa3ts3va7N0 LLUzww+Jt/Y/vBUFTYgdWyQdi7oh X-Google-Smtp-Source: APXvYqwf5X09uXyYPscu7c1OjunZbZTtxjv9B3zYL5jDDJaNtzuAzsOrKM/D12jMOSpieMi/XCiXZw== X-Received: by 2002:a05:600c:2115:: with SMTP id u21mr8259934wml.168.1569424840974; Wed, 25 Sep 2019 08:20:40 -0700 (PDT) Received: from localhost.localdomain ([109.227.33.147]) by smtp.gmail.com with ESMTPSA id a18sm12015723wrh.25.2019.09.25.08.20.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2019 08:20:39 -0700 (PDT) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Wed, 25 Sep 2019 17:20:32 +0200 Message-Id: <20190925152032.17848-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avcodec/qtrleenc: fix undefined behaviour X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Fixes #7991. Signed-off-by: Paul B Mahol --- libavcodec/qtrleenc.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/libavcodec/qtrleenc.c b/libavcodec/qtrleenc.c index cdd864bf82..6669c1302f 100644 --- a/libavcodec/qtrleenc.c +++ b/libavcodec/qtrleenc.c @@ -259,9 +259,10 @@ static void qtrle_encode_line(QtrleEncContext *s, const AVFrame *p, int line, ui /* These bulk costs increase every iteration */ lowest_bulk_cost += s->pixel_size; sec_lowest_bulk_cost += s->pixel_size; - - this_line -= s->pixel_size; - prev_line -= s->pixel_size; + if (this_line >= p->data[0] + s->pixel_size) + this_line -= s->pixel_size; + if (prev_line >= s->previous_frame->data[0] + s->pixel_size) + prev_line -= s->pixel_size; } /* Good! Now we have the best sequence for this line, let's output it. */