From patchwork Thu Apr 13 13:23:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Almer X-Patchwork-Id: 41143 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:4645:b0:e3:3194:9d20 with SMTP id eb5csp1244854pzb; Thu, 13 Apr 2023 06:24:09 -0700 (PDT) X-Google-Smtp-Source: AKy350Z6sCxKZ6elzVCL3xqaovZ0hfyNuxmMpLQUSS+rx6R+Qhqq/VV/R2QRp2JH3owZP+2nP5T4 X-Received: by 2002:aa7:cb49:0:b0:500:5627:a20a with SMTP id w9-20020aa7cb49000000b005005627a20amr2393254edt.25.1681392248732; Thu, 13 Apr 2023 06:24:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681392248; cv=none; d=google.com; s=arc-20160816; b=U4tkPtFZj50+88UQ4A317t+Kpp2/iIac7XtuI41dc2o6NGo6HQJEPVqqFHl5o4NEGX BF9HPalk73uHwTQyt3VBuM8hFtEtWKeYGFz+zVjrKSlF9aVgEvibXFlZ4Zh90VkV3rNx HWtcC4igT3Q0ClkRzXjwFQdyjmzRwxxQJHkuwRZNPaVtUJ6aW3R75j+dbEwH10xEy8pZ G2DpI1Cjyt6Gebkk37Rr1VSJr2ufunTAMfBHCOJ9xyYxOmSEo1dZofucgSJuBP+mDs+T o4D6VCyn4v9dLaOPbZRy7JZDbeIEOtWie3/5sa5EH9RpYolnTje+oB2f7UBpHCg/2gkU 0d1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:mime-version:message-id:date:to:from :dkim-signature:delivered-to; bh=0l8BNU2ce1pWzhDFQOOVMYDbDO/2po1vb0glX31MFtM=; b=09VCxSwDXwJuVs3v1DI+qsoJas9ZPdAvFZ0E+aH2OnsUTj9+Vhxuy6z7rJaRT4AJqu 7KP/hSpN1XvEqdfPBmzcGnl4Y6GiEAMlaASURWTtTnCU8QfiT+jD9WVpQ+WVkQOAVH2B gwKLbwnZajEbAxpEFX0sXoYlAk6/qB6yKDCXa6odwe1DQp1WOxjx3YixbxGjJo1isNpC +/1eSQ12bJBZXpFo/GuJMR8lvW7Puiw65n2QEZDCuopLDlYf0Re5NVZq2FX5tFm2ECtn FHX/dbzXHA7eN5TTiBg9tV1Jhk0GkL40trP3EwoMY+M4RN6iA4o0MC0VfuBIyH0IAqcQ lj2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=k6KBSta3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id bo23-20020a0564020b3700b00504b30a2291si1875039edb.121.2023.04.13.06.24.08; Thu, 13 Apr 2023 06:24:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20221208 header.b=k6KBSta3; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id AFA1E688373; Thu, 13 Apr 2023 16:24:05 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 812B6688373 for ; Thu, 13 Apr 2023 16:23:59 +0300 (EEST) Received: by mail-oi1-f174.google.com with SMTP id ec6so3195504oib.8 for ; Thu, 13 Apr 2023 06:23:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681392237; x=1683984237; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=boicIJo+TA0oi4l93L2JpGg97XhCLN2JiZdyZPTbCIc=; b=k6KBSta3f3zUSmQ+rHipAtjoIyXzcauAOEzH5SZJAnQrmBuLnw783pTTJxTnBM7CJm 0ziunStZlIrBWFd2vHc/BzgDlQLgALoF431wMLuKbmyduxfV4Kymr+DN7tlwqMpe0K+p ONReIA2MDU9zR02HFlGmB+unbKU6TB3tk8g6XXsfyYCfv/Tp/CIG4oYM85veq2kMFuO6 OTXbyX54ZRJDt2Wyhe/ayGM+orygUif9ZL7U+jXLp72i8gzfPmnWUwdEczYI7Lh43RJu D6gvuySFaUiUUN+mfznq3/hD/jCUvQZaGglEqs4Hdwj9HuONpU38ChmKRZrSO9mIYXlC S+iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681392237; x=1683984237; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=boicIJo+TA0oi4l93L2JpGg97XhCLN2JiZdyZPTbCIc=; b=Aj6Bh7S8xVUjezAjJTAHHcegVbmo6XqjQIYkggfjTy1UY9al5FMBRd4zpmoCNcHBM2 xhBofW2eJFNjX5W+SQb0HsQ+mQE02EYVQfqWdjIOh8fE56a6iWavFO2cWU9lzgzCUruM MVbTNgeRJ5oQ6gR+0u4b2GqhTnvGXNTJUpL16gJwVZyQZMbZ3AfNm3Qtlb0ekThi6gXU xzEXfVzzQc9JwECL8f/Uf4EiIAYcmKc1Dyz6otblYOyVJD9ciKecgU89gkCbJ5+W6/EN jSEBmjOazPenuEUwKcrENvLI0T/MAyrrsNBbwS4yZnnkbcGVZc2tA1ad1mzghswUnTxS dQXQ== X-Gm-Message-State: AAQBX9f8cqXDT/6UjridPsjwQTtIsJbTpIgXS6gMufOYD4gob31Us9bU y9MvfkRWDAxT4BbRWuaxB2GBEGq4geY= X-Received: by 2002:a54:488f:0:b0:386:c6ca:2b1e with SMTP id r15-20020a54488f000000b00386c6ca2b1emr1029164oic.7.1681392237550; Thu, 13 Apr 2023 06:23:57 -0700 (PDT) Received: from localhost.localdomain (host197.190-225-105.telecom.net.ar. [190.225.105.197]) by smtp.gmail.com with ESMTPSA id i189-20020acaeac6000000b00383ce81c186sm623220oih.32.2023.04.13.06.23.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 06:23:57 -0700 (PDT) From: James Almer To: ffmpeg-devel@ffmpeg.org Date: Thu, 13 Apr 2023 10:23:32 -0300 Message-Id: <20230413132332.3600-1-jamrial@gmail.com> X-Mailer: git-send-email 2.40.0 MIME-Version: 1.0 Subject: [FFmpeg-devel] [PATCH] avcodec/mp_cmp: reject invalid comparison function values X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: NLfU/qSo1Jck Fixes ticket #10318. Signed-off-by: James Almer --- libavcodec/dvenc.c | 4 +++- libavcodec/me_cmp.c | 9 +++++++-- libavcodec/me_cmp.h | 2 +- libavcodec/motion_est.c | 11 +++++++---- libavcodec/mpegvideo_enc.c | 6 ++++-- libavcodec/snowenc.c | 6 ++++-- 6 files changed, 26 insertions(+), 12 deletions(-) diff --git a/libavcodec/dvenc.c b/libavcodec/dvenc.c index 11dd5763af..cd442b524d 100644 --- a/libavcodec/dvenc.c +++ b/libavcodec/dvenc.c @@ -104,7 +104,9 @@ static av_cold int dvvideo_encode_init(AVCodecContext *avctx) ff_fdctdsp_init(&fdsp, avctx); ff_me_cmp_init(&mecc, avctx); ff_pixblockdsp_init(&pdsp, avctx); - ff_set_cmp(&mecc, mecc.ildct_cmp, avctx->ildct_cmp); + ret = ff_set_cmp(&mecc, mecc.ildct_cmp, avctx->ildct_cmp); + if (ret < 0) + return AVERROR(EINVAL); s->get_pixels = pdsp.get_pixels; s->ildct_cmp = mecc.ildct_cmp[5]; diff --git a/libavcodec/me_cmp.c b/libavcodec/me_cmp.c index e2f9f84b05..cd05e63ffd 100644 --- a/libavcodec/me_cmp.c +++ b/libavcodec/me_cmp.c @@ -473,8 +473,9 @@ static int zero_cmp(MpegEncContext *s, const uint8_t *a, const uint8_t *b, return 0; } -void ff_set_cmp(MECmpContext *c, me_cmp_func *cmp, int type) +int ff_set_cmp(MECmpContext *c, me_cmp_func *cmp, int type) { + int ret = 0; int i; memset(cmp, 0, sizeof(void *) * 6); @@ -533,9 +534,13 @@ void ff_set_cmp(MECmpContext *c, me_cmp_func *cmp, int type) #endif default: av_log(NULL, AV_LOG_ERROR, - "internal error in cmp function selection\n"); + "invalid cmp function selection\n"); + ret = -1; + break; } } + + return ret; } #define BUTTERFLY2(o1, o2, i1, i2) \ diff --git a/libavcodec/me_cmp.h b/libavcodec/me_cmp.h index 90ea76c891..aefd32a7dc 100644 --- a/libavcodec/me_cmp.h +++ b/libavcodec/me_cmp.h @@ -89,7 +89,7 @@ void ff_me_cmp_init_ppc(MECmpContext *c, AVCodecContext *avctx); void ff_me_cmp_init_x86(MECmpContext *c, AVCodecContext *avctx); void ff_me_cmp_init_mips(MECmpContext *c, AVCodecContext *avctx); -void ff_set_cmp(MECmpContext *c, me_cmp_func *cmp, int type); +int ff_set_cmp(MECmpContext *c, me_cmp_func *cmp, int type); void ff_dsputil_init_dwt(MECmpContext *c); diff --git a/libavcodec/motion_est.c b/libavcodec/motion_est.c index d17ffe42b4..df9d1befa8 100644 --- a/libavcodec/motion_est.c +++ b/libavcodec/motion_est.c @@ -309,6 +309,7 @@ int ff_init_me(MpegEncContext *s){ MotionEstContext * const c= &s->me; int cache_size= FFMIN(ME_MAP_SIZE>>ME_MAP_SHIFT, 1<avctx->dia_size)&255, FFABS(s->avctx->pre_dia_size)&255); + int ret; if(FFMIN(s->avctx->dia_size, s->avctx->pre_dia_size) < -FFMIN(ME_MAP_SIZE, MAX_SAB_SIZE)){ av_log(s->avctx, AV_LOG_ERROR, "ME_MAP size is too small for SAB diamond\n"); @@ -324,10 +325,12 @@ int ff_init_me(MpegEncContext *s){ av_log(s->avctx, AV_LOG_INFO, "ME_MAP size may be a little small for the selected diamond size\n"); } - ff_set_cmp(&s->mecc, s->mecc.me_pre_cmp, c->avctx->me_pre_cmp); - ff_set_cmp(&s->mecc, s->mecc.me_cmp, c->avctx->me_cmp); - ff_set_cmp(&s->mecc, s->mecc.me_sub_cmp, c->avctx->me_sub_cmp); - ff_set_cmp(&s->mecc, s->mecc.mb_cmp, c->avctx->mb_cmp); + ret = ff_set_cmp(&s->mecc, s->mecc.me_pre_cmp, c->avctx->me_pre_cmp); + ret |= ff_set_cmp(&s->mecc, s->mecc.me_cmp, c->avctx->me_cmp); + ret |= ff_set_cmp(&s->mecc, s->mecc.me_sub_cmp, c->avctx->me_sub_cmp); + ret |= ff_set_cmp(&s->mecc, s->mecc.mb_cmp, c->avctx->mb_cmp); + if (ret < 0) + return ret; c->flags = get_flags(c, 0, c->avctx->me_cmp &FF_CMP_CHROMA); c->sub_flags= get_flags(c, 0, c->avctx->me_sub_cmp&FF_CMP_CHROMA); diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c index 7d3c8875f2..bcd2f9ef1d 100644 --- a/libavcodec/mpegvideo_enc.c +++ b/libavcodec/mpegvideo_enc.c @@ -902,8 +902,10 @@ av_cold int ff_mpv_encode_init(AVCodecContext *avctx) s->quant_precision = 5; - ff_set_cmp(&s->mecc, s->mecc.ildct_cmp, avctx->ildct_cmp); - ff_set_cmp(&s->mecc, s->mecc.frame_skip_cmp, s->frame_skip_cmp); + ret = ff_set_cmp(&s->mecc, s->mecc.ildct_cmp, avctx->ildct_cmp); + ret |= ff_set_cmp(&s->mecc, s->mecc.frame_skip_cmp, s->frame_skip_cmp); + if (ret < 0) + return AVERROR(EINVAL); if (CONFIG_H263_ENCODER && s->out_format == FMT_H263) { ff_h263_encode_init(s); diff --git a/libavcodec/snowenc.c b/libavcodec/snowenc.c index 4cf7ff1145..31a48972a5 100644 --- a/libavcodec/snowenc.c +++ b/libavcodec/snowenc.c @@ -130,8 +130,10 @@ static av_cold int encode_init(AVCodecContext *avctx) if (ret) return ret; - ff_set_cmp(&s->mecc, s->mecc.me_cmp, s->avctx->me_cmp); - ff_set_cmp(&s->mecc, s->mecc.me_sub_cmp, s->avctx->me_sub_cmp); + ret = ff_set_cmp(&s->mecc, s->mecc.me_cmp, s->avctx->me_cmp); + ret |= ff_set_cmp(&s->mecc, s->mecc.me_sub_cmp, s->avctx->me_sub_cmp); + if (ret < 0) + return AVERROR(EINVAL); s->input_picture = av_frame_alloc(); if (!s->input_picture)