From patchwork Tue Sep 22 18:50:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul B Mahol X-Patchwork-Id: 22559 Return-Path: X-Original-To: patchwork@ffaux-bg.ffmpeg.org Delivered-To: patchwork@ffaux-bg.ffmpeg.org Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by ffaux.localdomain (Postfix) with ESMTP id E17DD44A2BA for ; Tue, 22 Sep 2020 21:51:18 +0300 (EEST) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id BD49A68B56A; Tue, 22 Sep 2020 21:51:18 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6558268B503 for ; Tue, 22 Sep 2020 21:51:12 +0300 (EEST) Received: by mail-ed1-f67.google.com with SMTP id j2so17181162eds.9 for ; Tue, 22 Sep 2020 11:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=zzAKYYSJOqePEd07kHHrW9zK4E4zwzBXUjwBuYtLS4I=; b=pqyN+TTxFKbajW8WPMXdvQZO426htovUFOAYRUMl4HXhQnhwmyY4QxX/ijjKc8s2Gp vFkB4mVhxZoiz72b79RxaPnpgALssPuDFqOzQIbDQx8FYCkgY07RoQCZJMBIzOuqu9pF chAep8aGrjIPrB8zMe5MpJTlRDx0o50/7Wl0HA48ZPIQDLF4EydcG2cizQsWagxZ8v0V oY+KbMfLq/O20dMC45YSsCcqa3vtA6vY8VSkS5J61ugKRP4DVDXnzpX5cU5Lc4ft4njU Fu9TlP77n258NoWj1Em9dE+45MG5fbu1fhEuVOli2rob3f67OoPmE06dxOuzkAVH5wOZ olog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=zzAKYYSJOqePEd07kHHrW9zK4E4zwzBXUjwBuYtLS4I=; b=Okg8JFwuQFz1jTfwMcCN10tOmGN6AmpRfbwcz0cNMS0qnN6o2X4h3l5Sf2fSjTeuHz T16GO81RAgeeazSXx1Y5UcjNuy1wwgEmobJ1+YlscasFhl9r5aU50bBOgrnEi8hFADbO JqY5Q3v+vLNf3Geuu0+aXYM4tH0Hwx8HMy4u6JwtKa3kPT69mn6nbFNXOxW8imNVZsZS iWAqWrQD3XgzU2qBL4kPnWw8vkmdHzkllYM+nSISu45IjIdEtOryAHTpgX53+rxCDSuA uPhoXEaQsu/0or+lhxZ2zkNdSBbCIhQnL+DVybnX5V+idGUCdZRHhvfCRcbx/MMuaph5 LGvA== X-Gm-Message-State: AOAM531L/J6Yo8XECUJVtxQj15NE0yk4GIJotH3idszKrSOEKc4h8ayI Kah4CluGK4qzvQoQhGqfeifET87QGNwNVg== X-Google-Smtp-Source: ABdhPJyhi9jnoUi7InxRs9VEl/8CVZlWmNu4Mg2pdE9tGCizG9+LpYgS7d9YYq6Knxgprjo2pSM5Gw== X-Received: by 2002:aa7:d58e:: with SMTP id r14mr5764804edq.52.1600800671026; Tue, 22 Sep 2020 11:51:11 -0700 (PDT) Received: from localhost.localdomain ([94.250.162.52]) by smtp.gmail.com with ESMTPSA id nm7sm11870505ejb.70.2020.09.22.11.51.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2020 11:51:10 -0700 (PDT) From: Paul B Mahol To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Sep 2020 20:50:57 +0200 Message-Id: <20200922185057.29345-1-onemda@gmail.com> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH] avcodec/cfhd: check that lowpass_height is >= 3 when used in vertical filter X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Also check for out of buffer access. Also return early when encountering fatal error. Signed-off-by: Paul B Mahol --- libavcodec/cfhd.c | 89 +++++++++++++++++++++++++++++------------------ 1 file changed, 56 insertions(+), 33 deletions(-) diff --git a/libavcodec/cfhd.c b/libavcodec/cfhd.c index ea35f03869..a2b9c7c76a 100644 --- a/libavcodec/cfhd.c +++ b/libavcodec/cfhd.c @@ -100,6 +100,8 @@ static void init_frame_defaults(CFHDContext *s) s->difference_coding = 0; s->frame_type = 0; s->sample_type = 0; + if (s->transform_type != 2) + s->transform_type = -1; init_plane_defaults(s); init_peak_table_defaults(s); } @@ -415,14 +417,14 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data > 4) { av_log(avctx, AV_LOG_ERROR, "Channel Count of %"PRIu16" is unsupported\n", data); ret = AVERROR_PATCHWELCOME; - break; + goto end; } } else if (tag == SubbandCount) { av_log(avctx, AV_LOG_DEBUG, "Subband Count: %"PRIu16"\n", data); if (data != SUBBAND_COUNT && data != SUBBAND_COUNT_3D) { av_log(avctx, AV_LOG_ERROR, "Subband Count of %"PRIu16" is unsupported\n", data); ret = AVERROR_PATCHWELCOME; - break; + goto end; } } else if (tag == ChannelNumber) { s->channel_num = data; @@ -430,7 +432,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (s->channel_num >= s->planes) { av_log(avctx, AV_LOG_ERROR, "Invalid channel number\n"); ret = AVERROR(EINVAL); - break; + goto end; } init_plane_defaults(s); } else if (tag == SubbandNumber) { @@ -442,22 +444,25 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, (s->transform_type == 2 && s->level >= DWT_LEVELS_3D)) { av_log(avctx, AV_LOG_ERROR, "Invalid level\n"); ret = AVERROR(EINVAL); - break; + goto end; } if (s->subband_num > 3) { av_log(avctx, AV_LOG_ERROR, "Invalid subband number\n"); ret = AVERROR(EINVAL); - break; + goto end; } } else if (tag == SubbandBand) { av_log(avctx, AV_LOG_DEBUG, "Subband number actual %"PRIu16"\n", data); - s->subband_num_actual = data; - if ((s->transform_type == 0 && s->subband_num_actual >= SUBBAND_COUNT) || - (s->transform_type == 2 && s->subband_num_actual >= SUBBAND_COUNT_3D && s->subband_num_actual != 255)) { + if ((s->transform_type == 0 && data >= SUBBAND_COUNT) || + (s->transform_type == 2 && data >= SUBBAND_COUNT_3D && data != 255)) { av_log(avctx, AV_LOG_ERROR, "Invalid subband number actual\n"); ret = AVERROR(EINVAL); - break; + goto end; } + if (s->transform_type == 0 || s->transform_type == 2) + s->subband_num_actual = data; + else + av_log(avctx, AV_LOG_WARNING, "Ignoring subband num actual %"PRIu16"\n", data); } else if (tag == LowpassPrecision) av_log(avctx, AV_LOG_DEBUG, "Lowpass precision bits: %"PRIu16"\n", data); else if (tag == Quantization) { @@ -471,7 +476,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (!data || data > 5) { av_log(avctx, AV_LOG_ERROR, "Invalid band encoding\n"); ret = AVERROR(EINVAL); - break; + goto end; } s->band_encoding = data; av_log(avctx, AV_LOG_DEBUG, "Encode Method for Subband %d : %x\n", s->subband_num_actual, data); @@ -489,14 +494,18 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data > 2) { av_log(avctx, AV_LOG_ERROR, "Invalid transform type\n"); ret = AVERROR(EINVAL); - break; + goto end; } else if (data == 1) { av_log(avctx, AV_LOG_ERROR, "unsupported transform type\n"); ret = AVERROR_PATCHWELCOME; - break; + goto end; + } + if (s->transform_type == -1) { + s->transform_type = data; + av_log(avctx, AV_LOG_DEBUG, "Transform type %"PRIu16"\n", data); + } else { + av_log(avctx, AV_LOG_DEBUG, "Ignoring additional transform type %"PRIu16"\n", data); } - s->transform_type = data; - av_log(avctx, AV_LOG_DEBUG, "Transform type %"PRIu16"\n", data); } else if (abstag >= 0x4000 && abstag <= 0x40ff) { if (abstag == 0x4001) s->peak.level = 0; @@ -510,7 +519,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data > bytestream2_get_bytes_left(&gb) / 4) { av_log(avctx, AV_LOG_ERROR, "too many values (%d)\n", data); ret = AVERROR_INVALIDDATA; - break; + goto end; } for (i = 0; i < data; i++) { uint32_t offset = bytestream2_get_be32(&gb); @@ -521,7 +530,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid highpass width\n"); ret = AVERROR(EINVAL); - break; + goto end; } s->plane[s->channel_num].band[s->level][s->subband_num].width = data; s->plane[s->channel_num].band[s->level][s->subband_num].stride = FFALIGN(data, 8); @@ -530,7 +539,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid highpass height\n"); ret = AVERROR(EINVAL); - break; + goto end; } s->plane[s->channel_num].band[s->level][s->subband_num].height = data; } else if (tag == BandWidth) { @@ -538,7 +547,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid highpass width2\n"); ret = AVERROR(EINVAL); - break; + goto end; } s->plane[s->channel_num].band[s->level][s->subband_num].width = data; s->plane[s->channel_num].band[s->level][s->subband_num].stride = FFALIGN(data, 8); @@ -547,7 +556,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (data < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid highpass height2\n"); ret = AVERROR(EINVAL); - break; + goto end; } s->plane[s->channel_num].band[s->level][s->subband_num].height = data; } else if (tag == InputFormat) { @@ -574,7 +583,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, if (!(data == 10 || data == 12)) { av_log(avctx, AV_LOG_ERROR, "Invalid bits per channel\n"); ret = AVERROR(EINVAL); - break; + goto end; } avctx->bits_per_raw_sample = s->bpc = data; } else if (tag == EncodedFormat) { @@ -590,7 +599,7 @@ static int cfhd_decode(AVCodecContext *avctx, void *data, int *got_frame, } else { avpriv_report_missing_feature(avctx, "Sample format of %"PRIu16, data); ret = AVERROR_PATCHWELCOME; - break; + goto end; } s->planes = data == 2 ? 4 : av_pix_fmt_count_planes(s->coded_format); } else if (tag == -DisplayHeight) { @@ -904,7 +913,8 @@ finish: } if (lowpass_height > s->plane[plane].band[0][0].a_height || lowpass_width > s->plane[plane].band[0][0].a_width || - !highpass_stride || s->plane[plane].band[0][1].width > s->plane[plane].band[0][1].a_width) { + !highpass_stride || s->plane[plane].band[0][1].width > s->plane[plane].band[0][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -944,7 +954,8 @@ finish: highpass_stride = s->plane[plane].band[1][1].stride; if (lowpass_height > s->plane[plane].band[1][1].a_height || lowpass_width > s->plane[plane].band[1][1].a_width || - !highpass_stride || s->plane[plane].band[1][1].width > s->plane[plane].band[1][1].a_width) { + !highpass_stride || s->plane[plane].band[1][1].width > s->plane[plane].band[1][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -982,7 +993,8 @@ finish: highpass_stride = s->plane[plane].band[2][1].stride; if (lowpass_height > s->plane[plane].band[2][1].a_height || lowpass_width > s->plane[plane].band[2][1].a_width || - !highpass_stride || s->plane[plane].band[2][1].width > s->plane[plane].band[2][1].a_width) { + !highpass_stride || s->plane[plane].band[2][1].width > s->plane[plane].band[2][1].a_width || + lowpass_height < 3 || lowpass_width < 3 || lowpass_width * 2 > s->plane[plane].width) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -1018,7 +1030,7 @@ finish: goto end; } - for (i = 0; i < lowpass_height * 2; i++) { + for (i = 0; i < s->plane[act_plane].height; i++) { dsp->horiz_filter_clip(dst, low, high, lowpass_width, s->bpc); if (avctx->pix_fmt == AV_PIX_FMT_GBRAP12 && act_plane == 3) process_alpha(dst, lowpass_width * 2); @@ -1042,7 +1054,7 @@ finish: dst = (int16_t *)pic->data[act_plane]; low = s->plane[plane].l_h[6]; high = s->plane[plane].l_h[7]; - for (i = 0; i < lowpass_height; i++) { + for (i = 0; i < s->plane[act_plane].height / 2; i++) { interlaced_vertical_filter(dst, low, high, lowpass_width * 2, pic->linesize[act_plane]/2, act_plane); low += output_stride * 2; high += output_stride * 2; @@ -1068,7 +1080,8 @@ finish: } if (lowpass_height > s->plane[plane].band[0][0].a_height || lowpass_width > s->plane[plane].band[0][0].a_width || - !highpass_stride || s->plane[plane].band[0][1].width > s->plane[plane].band[0][1].a_width) { + !highpass_stride || s->plane[plane].band[0][1].width > s->plane[plane].band[0][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -1106,7 +1119,8 @@ finish: highpass_stride = s->plane[plane].band[1][1].stride; if (lowpass_height > s->plane[plane].band[1][1].a_height || lowpass_width > s->plane[plane].band[1][1].a_width || - !highpass_stride || s->plane[plane].band[1][1].width > s->plane[plane].band[1][1].a_width) { + !highpass_stride || s->plane[plane].band[1][1].width > s->plane[plane].band[1][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -1158,7 +1172,8 @@ finish: av_log(avctx, AV_LOG_DEBUG, "temporal level %i %i %i %i\n", plane, lowpass_height, lowpass_width, highpass_stride); if (lowpass_height > s->plane[plane].band[4][1].a_height || lowpass_width > s->plane[plane].band[4][1].a_width || - !highpass_stride || s->plane[plane].band[4][1].width > s->plane[plane].band[4][1].a_width) { + !highpass_stride || s->plane[plane].band[4][1].width > s->plane[plane].band[4][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); ret = AVERROR(EINVAL); goto end; @@ -1214,7 +1229,7 @@ finish: low = s->plane[plane].l_h[6]; high = s->plane[plane].l_h[7]; - for (i = 0; i < lowpass_height * 2; i++) { + for (i = 0; i < s->plane[act_plane].height; i++) { dsp->horiz_filter_clip(dst, low, high, lowpass_width, s->bpc); low += output_stride; high += output_stride; @@ -1248,7 +1263,7 @@ finish: dst = (int16_t *)pic->data[act_plane]; low = s->plane[plane].l_h[6]; high = s->plane[plane].l_h[7]; - for (i = 0; i < lowpass_height; i++) { + for (i = 0; i < s->plane[act_plane].height / 2; i++) { interlaced_vertical_filter(dst, low, high, lowpass_width * 2, pic->linesize[act_plane]/2, act_plane); low += output_stride * 2; high += output_stride * 2; @@ -1277,6 +1292,14 @@ finish: output_stride = s->plane[plane].band[4][1].a_width; lowpass_width = s->plane[plane].band[4][1].width; + if (lowpass_height > s->plane[plane].band[4][1].a_height || lowpass_width > s->plane[plane].band[4][1].a_width || + s->plane[plane].band[4][1].width > s->plane[plane].band[4][1].a_width || + lowpass_width < 3 || lowpass_height < 3) { + av_log(avctx, AV_LOG_ERROR, "Invalid plane dimensions\n"); + ret = AVERROR(EINVAL); + goto end; + } + if (s->progressive) { dst = (int16_t *)pic->data[act_plane]; low = s->plane[plane].l_h[8]; @@ -1297,7 +1320,7 @@ finish: goto end; } - for (i = 0; i < lowpass_height * 2; i++) { + for (i = 0; i < s->plane[act_plane].height; i++) { dsp->horiz_filter_clip(dst, low, high, lowpass_width, s->bpc); low += output_stride; high += output_stride; @@ -1307,7 +1330,7 @@ finish: dst = (int16_t *)pic->data[act_plane]; low = s->plane[plane].l_h[8]; high = s->plane[plane].l_h[9]; - for (i = 0; i < lowpass_height; i++) { + for (i = 0; i < s->plane[act_plane].height / 2; i++) { interlaced_vertical_filter(dst, low, high, lowpass_width * 2, pic->linesize[act_plane]/2, act_plane); low += output_stride * 2; high += output_stride * 2;