From patchwork Tue Sep 12 13:26:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Frederic Devernay X-Patchwork-Id: 5129 Delivered-To: ffmpegpatchwork@gmail.com Received: by 10.2.36.26 with SMTP id f26csp607286jaa; Tue, 12 Sep 2017 23:50:11 -0700 (PDT) X-Received: by 10.28.16.16 with SMTP id 16mr1710279wmq.62.1505285411769; Tue, 12 Sep 2017 23:50:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505285411; cv=none; d=google.com; s=arc-20160816; b=QC11qCOEC413DbWmlNtj4dsXsNCvOupv3NdUjKYeFfYnqfyjkdBGOzonPYx4wGW05B H44f6Ze+0tqnB+hSp5CFh8gXdCZ2tl6OJVsV6M0nxJcqt6OI0bOe9WrF/N3iWv6QRXQk baV1dhXLg5ok1vbj0jb0Mwr7hGcGQnUnE5IhZgcPnnYV9vf1uuR6PXlLEENvoPLT13fB ye5CzjqyvCQsaoSwdz75DY9OggO4nmdoTD2sJplP1vIb03tnLMJy/HgCAUaAMibQsB9V 45frWyje0JYLzUt+MnKioh4T5ovuHE2+Y0EGWULicVCkpESHzwKwG9k0LQ9g1K2vwzFJ yUeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to:arc-authentication-results; bh=szuzDV8MZlPPqDQxuuQ4TBX7g8kV2wMeADezjsFI+zE=; b=rQokmuGFa/+93fw6qoSQAK/Gf6ej2Ms+URtE7SkrkMPgiSdQQCmvT3b2uLYuvaHhei u+xrfVoaG0Bw2DzzsD20RQ9FMigJ3gOEkqn6UI94M8Tk/hFrQ03Vlr/gwpnHKo02qs90 iIYrmsDRc1raaO6prLMKhDOkDvQq+1SwLVn61ZGldewQQZOuFplc5vrMVtme7f7RcgOT 8eDdQ/MQ0BM+0t52Sj66AgO3ElDgjDooqjcgKpIzZxVeUzrcqqvHlanRxF8GIkZ3UTso NDP48MiR9ivtJakqaif2ycCHRhRL0Puf8A+h4ZwyXAf+ApSPA3Mw9vFT37XcSjGhNcrv xj2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YQmqr9eY; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id 75si488767wmy.174.2017.09.12.23.50.11; Tue, 12 Sep 2017 23:50:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=YQmqr9eY; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 1E450689C09; Wed, 13 Sep 2017 09:50:01 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 95482689BBF for ; Tue, 12 Sep 2017 16:34:08 +0300 (EEST) Received: by mail-wr0-f193.google.com with SMTP id n64so6210770wrb.2 for ; Tue, 12 Sep 2017 06:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ElPSZMx/8ALJoU2jJSkgTohjakj/PEsV1XaPcSoPZ5o=; b=YQmqr9eYl0u1Tvj43wuMAqbNSVi8jTixMT/nYgqaDl1E35HY4OJKSsegPPLJMQArHD zyBC+Zz0kgiVcfQztgSaPHMOTZmPUPxAuhH1815QxHk2cH1u6S0yvVFRYpmIdl9Qv+0x JQmMslsHED/wdpkjnoh4DqlnMhnv1u0AZBihfng05XP9SMXwHLdhVnIkAE4QSOQkzP0x VO/caujCpT7wDHW5uVYHGjhyJ3QzJ3WIGdqbQ/5k1N8xE/WNnaSGmaxC3wt69FqhA5H8 F65yxItFpCFiqvf2RnrGTe8FCIm5ZtId+UIOo7oNb/Drqw0vBJt3c+GKBbE1W3IREoFg /9gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=ElPSZMx/8ALJoU2jJSkgTohjakj/PEsV1XaPcSoPZ5o=; b=mK3Rte448QTTjQnswjw/w8v2vW6g1YU6RKAzKIm5QhjoAeBSsiK0ty1yDLidrxFxOP IH/zKiTRNUnLlGi8G7STbMM867TeWR9nhZoeab9WMhqZzstUwDT1VMrBeqIy8vxGWRQA /dDHflG70tYhmXfyILxaf61SduHybgbYcnZKC2kQUSsxllIsYbE5z9ccGlCMxVa3xWCZ 8k+OQlN9g4dzbWNM0got4fy9Xdm2LJUVCiEuVCQ0TezXofLT36Z1CkNMaP9HfZGyFvhZ LMmvhPRBcAsQ3QgNU8xPZNSrGXb1OjdJP+NUzYssfDAAunuOX/90GiECgqN+t+MSKX6S 89xQ== X-Gm-Message-State: AHPjjUgK5Pyzce3viVnQFPClXBxtQPnQYX7yolGOKsemDTTfmy+OLmpJ b2FWJAjcX5avNWgw X-Google-Smtp-Source: ADKCNb508wxlcakJngNRSfxORV6GPHrvc9AGtqfgW0KeQUikij7a8r+pYL3RxT9wuTAe5r51X+t82w== X-Received: by 10.223.134.231 with SMTP id 36mr13364069wry.141.1505222782044; Tue, 12 Sep 2017 06:26:22 -0700 (PDT) Received: from raiponce.inrialpes.fr (raiponce.inrialpes.fr. [194.199.26.5]) by smtp.gmail.com with ESMTPSA id r72sm11719912wmb.9.2017.09.12.06.26.19 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 12 Sep 2017 06:26:20 -0700 (PDT) From: Frederic Devernay X-Google-Original-From: Frederic Devernay To: ffmpeg-devel@ffmpeg.org Date: Tue, 12 Sep 2017 15:26:15 +0200 Message-Id: <20170912132615.10975-1-frederic.devernay@m4x.org> X-Mailer: git-send-email 2.14.1 MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 13 Sep 2017 09:49:59 +0300 Subject: [FFmpeg-devel] [PATCH] avcodec/dnxhdenc: fix DNxHR 444 encoding crashes see https://trac.ffmpeg.org/ticket/6649 X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20Devernay?= Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" From: Frédéric Devernay --- libavcodec/dnxhdenc.c | 16 ++++++++-------- libavcodec/dnxhdenc.h | 4 ++-- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/libavcodec/dnxhdenc.c b/libavcodec/dnxhdenc.c index 5a0e6de6a5..0d80381a2d 100644 --- a/libavcodec/dnxhdenc.c +++ b/libavcodec/dnxhdenc.c @@ -749,14 +749,14 @@ void dnxhd_get_blocks(DNXHDEncContext *ctx, int mb_x, int mb_y) ptr_y = &ctx->edge_buf_y[0]; ptr_u = &ctx->edge_buf_uv[0][0]; ptr_v = &ctx->edge_buf_uv[1][0]; - } else if (ctx->bit_depth == 10 && vdsp->emulated_edge_mc && ((mb_x << 3) + 8 > ctx->m.avctx->width || - (mb_y << 3) + 8 > ctx->m.avctx->height)) { - int y_w = ctx->m.avctx->width - (mb_x << 3); - int y_h = ctx->m.avctx->height - (mb_y << 3); + } else if (ctx->bit_depth == 10 && vdsp->emulated_edge_mc && ((mb_x << 4) + 16 > ctx->m.avctx->width || + (mb_y << 4) + 16 > ctx->m.avctx->height)) { + int y_w = ctx->m.avctx->width - (mb_x << 4); + int y_h = ctx->m.avctx->height - (mb_y << 4); int uv_w = ctx->is_444 ? y_w : (y_w + 1) / 2; int uv_h = y_h; - linesize = 16; - uvlinesize = 8 + 8 * ctx->is_444; + linesize = 32; + uvlinesize = 16 + 16 * ctx->is_444; vdsp->emulated_edge_mc(&ctx->edge_buf_y[0], ptr_y, linesize, ctx->m.linesize, @@ -771,8 +771,8 @@ void dnxhd_get_blocks(DNXHDEncContext *ctx, int mb_x, int mb_y) uvlinesize / 2, 16, 0, 0, uv_w, uv_h); - dct_y_offset = bw * linesize; - dct_uv_offset = bw * uvlinesize; + dct_y_offset = bw * linesize / 2; + dct_uv_offset = bw * uvlinesize / 2; ptr_y = &ctx->edge_buf_y[0]; ptr_u = &ctx->edge_buf_uv[0][0]; ptr_v = &ctx->edge_buf_uv[1][0]; diff --git a/libavcodec/dnxhdenc.h b/libavcodec/dnxhdenc.h index 9b43f6e9c6..26c3eec695 100644 --- a/libavcodec/dnxhdenc.h +++ b/libavcodec/dnxhdenc.h @@ -75,8 +75,8 @@ typedef struct DNXHDEncContext { int intra_quant_bias; DECLARE_ALIGNED(16, int16_t, blocks)[12][64]; - DECLARE_ALIGNED(16, uint8_t, edge_buf_y)[256]; - DECLARE_ALIGNED(16, uint8_t, edge_buf_uv)[2][256]; + DECLARE_ALIGNED(16, uint8_t, edge_buf_y)[512]; // has to hold 16x16 uint16 when depth=10 + DECLARE_ALIGNED(16, uint8_t, edge_buf_uv)[2][512]; // has to hold 16x16 uint16_t when depth=10 int (*qmatrix_c) [64]; int (*qmatrix_l) [64];