From patchwork Fri Dec 28 20:30:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Mogurenko X-Patchwork-Id: 11572 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id D5FCD44C781 for ; Fri, 28 Dec 2018 22:30:26 +0200 (EET) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8C8B268A708; Fri, 28 Dec 2018 22:30:23 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-lf1-f67.google.com (mail-lf1-f67.google.com [209.85.167.67]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id EF20068A6A5 for ; Fri, 28 Dec 2018 22:30:17 +0200 (EET) Received: by mail-lf1-f67.google.com with SMTP id z13so15193523lfe.11 for ; Fri, 28 Dec 2018 12:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=1wQhYLdyAZqKwz8RdEzXcET8ZSu0vPid9/JdSX2Huc4=; b=S175sNXZujSVtqb/TBUR3LjC/dXtgvT2qVkO7s4wuE7NPAqssbdgdipTKIj1WRbEZS tAulr0xBLKXzvEM/WbKoQib1qpQYHh9eUciDU0VPkyJ0HyJEZdkhtlMeXPiDMFyNaF5Y HB311BaQ+iDXP5OPkBVDR3XtCq4Ye96afvujEFlbRXMYuyms6RWMmgpd4IDQaDBuRV6/ shaSxHhRYvR0DNnVuGNiB+h+0zbrieXMo5tfC6CuBbD8BroJMrlPcj9atldDBLdZU1U/ /+nmcLIgiQ0IZi4y7aufX4gDq6Pto1dA7XStCNLibSiQ3moGooTBkj5ibJAsT2S7U5bp zinQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :mime-version:content-transfer-encoding; bh=1wQhYLdyAZqKwz8RdEzXcET8ZSu0vPid9/JdSX2Huc4=; b=RzGe+QhmbuatqFSSkA4dom8gjLAh7XoS/C3+tk5QzvaAfhRW37UJUnkju6ZBe6mv7d UNq7QpxG0W1vaah0QlsF8jIw5yn7C4175voQXlNcyDL9wNeEUMYkq9SwPGGFQbFtNXiS 4vQR/lOfVrLg6Ivv92hibAJ7exkgciOSAkXJRuAt5dvE1hYEEpZzjZ6N4FUENegUZ5Ak XalIkgPwJ7ZT2xjdRh2awJrk6GPH+cVk5pbs3fWiYseEC7aPYY7+Y8y5fPPko6R+RuQM mn5yULwRGmm8fCt7TABBQC6EE+WYvlYXy/rxludJQs9RQv85sugB0KpvsqAoaMgcbtph tNOQ== X-Gm-Message-State: AA+aEWYEO+pbRlL/6kLoETzYBjtBh8JA8s1uYPdU1XsKJ48tTPmMY5x1 CkqAIYWqrY9xu7XxUN2f4VboCQKY X-Google-Smtp-Source: AFSGD/XB0Eve2Atnrb/40qtRtCDvqDij4OpmR2JCWWhzFR6mb5DniGTpq9KJrw87Z4i2C7sJE0Mptg== X-Received: by 2002:a19:24c6:: with SMTP id k189mr15312794lfk.77.1546029023344; Fri, 28 Dec 2018 12:30:23 -0800 (PST) Received: from localhost.localdomain ([77.121.132.178]) by smtp.gmail.com with ESMTPSA id j12-v6sm8672944ljh.66.2018.12.28.12.30.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 12:30:22 -0800 (PST) From: Alex Mogurenko To: ffmpeg-devel@ffmpeg.org Date: Fri, 28 Dec 2018 22:30:08 +0200 Message-Id: <20181228203008.26892-1-alex@mogurenko.com> X-Mailer: git-send-email 2.19.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/prores_ks: fixed luma quantize if q >= MAX_STORED_Q X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" problem occurs in slice quant estimation and slice encoding. if slice quant >= MAX_STORED_Q we dont use pre-calculated quant matrices but generate new: qmat = ctx->custom_q; qmat_chroma = ctx->custom_q; for (i = 0; i < 64; i++) { qmat[i] = ctx->quant_mat[i] * quant; qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant; } as you see qmat and qmat_chroma both point to same ctx->custom_q result it will contain chroma qunatizers as qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant; last in the loop, after all we pass qmat/qmat_chroma to function where we estimate encoded slice size or encode slice: estimate_slice_plane / encode_slice_plane custom_chroma_q was added same way as custom_q e.g to encoder context I would prefer to have it on stack thought --- libavcodec/proresenc_kostya.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c index 9a77d24fb6..e045a972f1 100644 --- a/libavcodec/proresenc_kostya.c +++ b/libavcodec/proresenc_kostya.c @@ -222,6 +222,7 @@ typedef struct ProresThreadData { DECLARE_ALIGNED(16, int16_t, blocks)[MAX_PLANES][64 * 4 * MAX_MBS_PER_SLICE]; DECLARE_ALIGNED(16, uint16_t, emu_buf)[16 * 16]; int16_t custom_q[64]; + int16_t custom_chroma_q[64]; struct TrellisNode *nodes; } ProresThreadData; @@ -232,6 +233,7 @@ typedef struct ProresContext { int16_t quants[MAX_STORED_Q][64]; int16_t quants_chroma[MAX_STORED_Q][64]; int16_t custom_q[64]; + int16_t custom_chroma_q[64]; const uint8_t *quant_mat; const uint8_t *quant_chroma_mat; const uint8_t *scantable; @@ -574,7 +576,7 @@ static int encode_slice(AVCodecContext *avctx, const AVFrame *pic, qmat_chroma = ctx->quants_chroma[quant]; } else { qmat = ctx->custom_q; - qmat_chroma = ctx->custom_q; + qmat_chroma = ctx->custom_chroma_q; for (i = 0; i < 64; i++) { qmat[i] = ctx->quant_mat[i] * quant; qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant; @@ -902,7 +904,7 @@ static int find_slice_quant(AVCodecContext *avctx, qmat_chroma = ctx->quants_chroma[q]; } else { qmat = td->custom_q; - qmat_chroma = td->custom_q; + qmat_chroma = td->custom_chroma_q; for (i = 0; i < 64; i++) { qmat[i] = ctx->quant_mat[i] * q; qmat_chroma[i] = ctx->quant_chroma_mat[i] * q;