From patchwork Wed Sep 28 16:35:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 38446 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:3b1c:b0:96:9ee8:5cfd with SMTP id c28csp358236pzh; Wed, 28 Sep 2022 09:36:39 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6h0CD/LYJd9jEmWi7eAqOjaHDBmiS+J5AsuzNpsIvCQt5LvEiOaOephZlnLD+MCp4+w0WS X-Received: by 2002:a17:906:4796:b0:787:7693:7942 with SMTP id cw22-20020a170906479600b0078776937942mr6083326ejc.752.1664382999273; Wed, 28 Sep 2022 09:36:39 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id kl13-20020a170907994d00b0077771b6d988si4593434ejc.558.2022.09.28.09.36.38; Wed, 28 Sep 2022 09:36:39 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=oMZPPrxo; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 8C86068BBA4; Wed, 28 Sep 2022 19:36:32 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05olkn2014.outbound.protection.outlook.com [40.92.89.14]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id A695F68BBC2 for ; Wed, 28 Sep 2022 19:36:25 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A6rERz3iC56dVIK+kcpwJPQZ9zIZydnxb2z+tIqpOn31WlMjrVp+CDnB3p7WFsNwZQRbql8LCS/M8rKfBahhAB2luSs4+aw0UlC3GGWKyZz31961u1x1LYshkMABjr2CYE0W8Q8jGEG7f24CcIjuPB99tembGSZbtOM4NptSQ5dQveojVxm8aDEDc4L3Ug9WKr7qcROIJq6Aj4b3bAtPAWMQCVPmbuOzeoZUrqc+Q944Qj6jTRqm0lRzTw+qBaW2kQwqP/QtwKi7bPnwHEM7dqmYxX81EIQ81onG87eVt0303y9dzmbChqU9rFbp/l0IgC0o7rq+vy+RyVqmtRuDag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LbxS0mHZhYwjRcUGmTq1AjOj62t2rtSLQymRet6/5hs=; b=fFTYqp9zfuFzwg/7YYOzFTmZU8zFOwWVwfWXN7ynPLLzocBjVK8F9jYVcK0JZOa8q/lBiENXC8GM1kSUfZiWYLd8diFs9tD6lnelpx73Cwss/TL4yabt9nmlq0s28KulYNMTFzPJ4ITSYgs+EDhm8HLRWR3c9LT4panGFhZODtxb83Tz11+ju7sjDQACngRJNifS60wPbwobaj7FtQkusJi4/7jgBFPd9f6/fiiPtDBHdT+65MkQOTeMfbBxY0qGh70W3t6kIz6zLkESIGFCkKwpsbHux49W5fv8oOHY2VD/GbcHIgh0mpzNW7E4DhhCF+hyTvGxQiCtdFpN9H4Hig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LbxS0mHZhYwjRcUGmTq1AjOj62t2rtSLQymRet6/5hs=; b=oMZPPrxoh8H9tWBEgHN1Wpykpt+BBILXtlSQo6FsyEglecPoA4hRpds++KrYRmkay+VG4CE/CZ9b8Uz5y6bRmMtIcH1yKFdP6ZRH8ks+fq9THfO9M4q4rLPefKeseAdTY14l+whJ1tlTokM/q4I94FpsrOsN8a+/E/xDgG/nxMMQTJhNzxxY+D65OCRFY+i5JxAXjBVUjaPACXiRD/1kabSQOGnsm7XpLsE6M27RpEAKcHAu1pJjjCM64iJgibjzRF/mKcemCaNakeqHYmt6mGPfy0oc//jvAgV7MU1PmvB/6nsqUf2ldcuvsVarWO+LqaHkm5wbpBgY2x2/AZPhsg== Received: from GV1P250MB0737.EURP250.PROD.OUTLOOK.COM (2603:10a6:150:8e::17) by AS8P250MB0006.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:359::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.15; Wed, 28 Sep 2022 16:36:15 +0000 Received: from GV1P250MB0737.EURP250.PROD.OUTLOOK.COM ([fe80::68bd:2fc7:ac52:38f8]) by GV1P250MB0737.EURP250.PROD.OUTLOOK.COM ([fe80::68bd:2fc7:ac52:38f8%9]) with mapi id 15.20.5654.025; Wed, 28 Sep 2022 16:36:15 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Wed, 28 Sep 2022 18:35:49 +0200 Message-ID: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: X-TMN: [hy7z9n5vKDgtSpwbVcQcSxdcUN938lwD3++12lqOE4k=] X-ClientProxiedBy: ZRAP278CA0009.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:10::19) To GV1P250MB0737.EURP250.PROD.OUTLOOK.COM (2603:10a6:150:8e::17) X-Microsoft-Original-Message-ID: <20220928163550.916073-3-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: GV1P250MB0737:EE_|AS8P250MB0006:EE_ X-MS-Office365-Filtering-Correlation-Id: 9690e42b-504a-4e8e-f3ef-08daa16f8fcb X-MS-Exchange-SLBlob-MailProps: AZnQBsB9XmpPcoZqFl/i/jhf7c/pX34TsjZJlEjt0SXfcLXknORsNAG1XV5XqP31J3dRSUascec8hoJt28YgTlx9BhqkFuWBaBlzqXGuB7FC70mgYTkj6Dz2j7q3ya/NuxDEh1xJpGfiriRdQzm4BSzRkcPzLgAhoo9oSBnsEsvuVZhm16vl8l9oprdWsegVl6PEVJtI3dR62pbZOyr8A1CdJp5iqYCua4QkceRbjBcld7DucwXnW6gqh8ZVIrGVQ2JsHUGpaNcaSodUtF8EdpnLTdV2blfCUo1uEkFwuOzdvBGj+KrM7fQj6s7uWc7H10SI53fSJuDVuKY2R6W+QgfU5Dimy2uhpRI/3DEj1/b2eSsKvwmlPl8xWDuk4VfhLF5EJQpXWvdBaltF6H5wOgN9icZ+/bZwpXoz8PKfmK2bxQEFu12z5gUw8+/G7RBFPoKqKPkDDSxDuyw5Yz4n5C2ug3o7j8lu8EBgr/4+ox5OuWR2N3FySYr+EOp5WE3sd5bPcDT9dnWoMh1mTinwX0dUJrkiff1czuNpbO35K0t0xDbWxoBTQrdR7n7GW0BMC9uBPB2whGdvdhfsx4oRwLp5hzt3SFxCrqJBu9UVJVWW1lTrm7Pdt6xTmQ6NTbWKeiOKccbFYlXuw1kSGi4GqpQoO5dIC70mI45RvkOGQhBirYUnISJBTV71JD3JRM3r3EDPwgkufomviyD3syIBNUstxJSzn5Y4UHSEu2wtrIfyHljt8C3Mjupn7a0PyfAbag4Q757wqNE= X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: K/n0cDsphfCtCJF38X3HQG3hqIz3JysPDAL6bW65MUNZ5zLEeOg0K5ubaWtvPcX+g66kGFCjab4big2OlVuuleJmk25ZGNY1CKOmg4OV4sbGZPQ3UyqCYtPOuIPPLIxq9Fz9fvJHGXod9sE+dEq9EQif9qa7xMQyccZIbyd9LQ4+4n0lN44/++NbS1qNqA0FkoBwPJ7pldfbB4holhD4WED/wOWBlrtZmk1KWiyugBNgN70r7QJYXKJDmGC5PgqN0iG2BwiStz25EizLI+muMkRF2Gz4AzwWcIvd7M4p3UkP6sJUiyc+83BX9FFCm3HfMnW1BR23nILVGvY2tXRprMhzVrw8kDFmCbThltQCrDzhvktjGPJkXXynFajtgDVXk/osLPXpMntGFNY0TGLrsgfKx5iLS5yWEboRGlh6hAcCtk7vt+PGWFzEADUqg+VU/ZVYPFqSA7nVF7cyfmlqmbEvxvMtUEsgUPN7aMfoDNnIW06WgJ2KF+62AVfIPNtXHUy3AASPF3Fxu0XIkyEf9FVMk8tFUqarFHsE79DELOjyC/SkjcboyhEvbCyfBxwLl9O9pACTl5V3TjQl4reLZnU1t+TmMYO+SUK+rl9mURLZBp0cFHOXvayClNjOJAHNP1syE+Eg97nQl82jXLspNg== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fIHWc3FI42/94SH3FRJ7tiQr/mGFCIr9qc3VW2QD2cLDwmqn3Hh2NzE10Z3RlvN2sehaPIYw0wV4zexwJYZJljlnKpLMabXHQ7yBoejQtukpQUP7udyfntPZeUa1fSaWLheLr7mKNaj9qMsfQWDIkoQKSTecBXWZTiuKtZWQsKktAhD/LHAvbDgjkZOQiDRv6KsIedT5RWu1qn2Dj6kA/TJDe9QawQFy2VnM4QPCc7jAxSqeC6GA/gbKPoIOJkrWyH147fKD/L/Hp17A/7fvCVTxNQU4HnmyX085akHB5ledR8WV45NHCJWL0hlnTEiBIougPxW3I+r1RVBD96rm4yhoVgL0FQHu4hfQJC9HkhG/lnAoycUK6Iy9V5tx5sVww9NS0U51gY8l7P5D6bitilr4UpzD2sy5p4Qwm2bO+TEV2Wg9Yskj9Bekid8KmsiWLDsTkI0u/TgOGINSaczrRbxlPQazt7oQJowt3SFuQN1u+Noji9WuFrN7SAwQtC6KsNLVZxL1gWaA9oz+Wr+WcddiBYq7bB9b/3AySmUbh/Wvh5MXJN7RVlh2otjRnVJ9Kdz9bEezPWM6oTZj3/GexldHmEWL4u+C6YD/15tZssKZf69eD2kJq1yfkreXYtDkg6/ro3lAfiRqrp3yLaI/F9r9F7MOBdwaXEWJwN5d+H0HhfzZLZwE2SiTiM4ADGSgtD9wlySr0zJgoWhP35ihOevyinjRvLUWnA7CKCAEOJQrLY9usohHIRd6XGLe2cIuSYlPYEuiRzkUIO7ZhelBMFoyikhCsJbbElE1O2JKVmk3cjUnoIdXyhne5DjwQcWJoThNc0xM0ScYMzdDGp8IL+YBNe14mT0rC/Ut5EZVZ0Vomw+yVcOdgZq6q12ELVA2ZMpgwlbHxkFydoxsdjsruVJXofFYa21M3w+VoI97AGMvzGNKulpzpwbqBlN/vsf7FUJVJAix9LwHvni2saEmUnqtxvsvX4qgIsTjOV4RR52gVKrNYli6HnNZX8j18fMPKZrwGZH1kPCjALLDLWbK7lObkcVzdBP4Ut6c26qT4CGgkymPaEl2FXQoAAJmo8GL8YuzDxqe9dwa/43Js3o7L4mi1zfWoH1ob0w3sPEj2jkZGbLeCu1er3HrcPNjiM2hcUdzHhrV+XHRbv5xkNfKF+hmEGMwp4U/ueYTrdIUaCvj+N5jIRxDYaC/L0I0cpRsMJ5fISRPWsDnET8lyyS8wnftHzqUGiBLJsnnpGpcHZhS5ZaVtL4fDE0UTXx0dLSGg/8d+QVs9HV0pjQQA55FwN1zicG9/kALBC5t8lMOH69skvPg8K+Z3ccVPwf2cIaC X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9690e42b-504a-4e8e-f3ef-08daa16f8fcb X-MS-Exchange-CrossTenant-AuthSource: GV1P250MB0737.EURP250.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2022 16:36:15.4171 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8P250MB0006 Subject: [FFmpeg-devel] [PATCH 4/5] avcodec/rl2: Fix undefined pointer arithmetic X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: /2D8DF7xoPA2 Don't increment back_frame if it does not correspond to a real buffer. To do this, handle copying from the back frame separately from the "use coded value" codepath; also use memcpy for the former, as the chunks here are typically worth it. Signed-off-by: Andreas Rheinhardt --- libavcodec/rl2.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c index 76982f0426..7938ef1d92 100644 --- a/libavcodec/rl2.c +++ b/libavcodec/rl2.c @@ -69,13 +69,16 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, uint8_t *line_end; /** copy start of the background frame */ + if (s->back_frame) { for (i = 0; i <= base_y; i++) { - if (s->back_frame) memcpy(out, back_frame, s->avctx->width); out += stride; back_frame += s->avctx->width; } back_frame += base_x - s->avctx->width; + } else { + out += stride * (base_y + 1); + } line_end = out - stride_adj; out += base_x - stride; @@ -89,16 +92,32 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, len = *in++; if (!len) break; + val &= 0x7F; } - if (s->back_frame) + if (back_frame) { + if (!val) { + do { + size_t copy = FFMIN(line_end - out, len); + memcpy(out, back_frame, copy); + out += copy; + back_frame += copy; + len -= copy; + if (out == line_end) { + if (out == out_end) + return; + out += stride_adj; + line_end += stride; + } + } while (len > 0); + continue; + } + back_frame += len; val |= 0x80; - else - val &= ~0x80; + } while (len--) { - *out++ = (val == 0x80) ? *back_frame : val; - back_frame++; + *out++ = val; if (out == line_end) { if (out == out_end) return; @@ -164,7 +183,9 @@ static av_cold int rl2_decode_init(AVCodecContext *avctx) back_size = avctx->extradata_size - EXTRADATA1_SIZE; if (back_size > 0) { - uint8_t *back_frame = av_mallocz(avctx->width*avctx->height); + /* The 254 are padding to ensure that pointer arithmetic stays within + * the buffer. */ + uint8_t *back_frame = av_mallocz(avctx->width * avctx->height + 254); if (!back_frame) return AVERROR(ENOMEM); rl2_rle_decode(s, avctx->extradata + EXTRADATA1_SIZE, back_size,