From patchwork Sun Sep 26 06:32:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 30581 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:6506:0:0:0:0:0 with SMTP id z6csp3042800iob; Sat, 25 Sep 2021 23:33:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyHJ/C4SZ93/4jc5AAn+PT96jI91A03AVOKtc5gYSJyg7+ugwBM8tpzRBYBDtiUShL4MF1E X-Received: by 2002:a50:c006:: with SMTP id r6mr16045564edb.289.1632637988615; Sat, 25 Sep 2021 23:33:08 -0700 (PDT) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id e3si14922065ejm.466.2021.09.25.23.33.07; Sat, 25 Sep 2021 23:33:08 -0700 (PDT) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b="f7m/8IBx"; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 375F268A901; Sun, 26 Sep 2021 09:33:04 +0300 (EEST) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-oln040092075045.outbound.protection.outlook.com [40.92.75.45]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 2824F68A831 for ; Sun, 26 Sep 2021 09:32:57 +0300 (EEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C8VdF7VjKR3AWTSubl8IMY3ryzl9UWUgqfiOe99FK8WGK/yeZgllGrTzT1GYEYUmf7cnhFrZs546VkcSSPgBOCzTiVBRr3l+MZiVsB45PeIhToU+SmxMGpXN7EtjDOTVHcwydmmnZgMkI2dz7E6qRQhJCwu1MfApVke4i+5kWryAr9lZdKMsCv59s9Wp5omSr5k5tp5rOt+jrqIk5W7/clO7Q6KeoS9NP0c+3YBBK0TP087tnokTAr+fPZ0zvFnz60eaQIjyCQv2Au0Ot41YzRvij/4s2wPLGBQiJDf1CPq7GzW2ui23DW5vDtKBl3VqBOpp07cnSSubOpuuSbtprA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=AZFvjhHyzETFT8O5vRqKUDLc4rZPBQAbUAfvMlfTwcc=; b=FCvSnb3T0qBsFg2Hb1Qp5jwvuy9YuWljLsg4ONzH4gg60BXwWUvvhQIAQicwRtvFQYlPju3ax8KTQ58sQcburpWSQp7tOQn/IlgcLaF4SzS0aHMySZ02AeQbdeLVtmGc9YNS6MaSya6akXkEI4mEkMXuyriDUpuWyYiFYuO55bxV5C8K6T3k/O2dYhXxQCrMwaik4zuZZvb4kttbpYYejmGyCLyOrUKyi95dz964CHYd/iFAwjxaucqib6YTOfwuiRE9k81Gg70DfsgMe40NhcNqPQQ6aBnYeqL2RxQ9ImOSiJ9hHqKi/V6bTNHHi8LHwTbf9xcbWYqIhmeVBVuP4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AZFvjhHyzETFT8O5vRqKUDLc4rZPBQAbUAfvMlfTwcc=; b=f7m/8IBxJ8GQLZaMdV9XwWIoRqrgZt4dAFj/XGdpd85qhlIqAGKb59stEswMo84k3YBFJh39NvUlBD5m4JSdCOhCpd+ExU2FcdEmI6UfiYpWZWuHwdcSb/1NjseuJH7dIkv3069cOLi81PGlFwLlJDqioKqctVtD2ndt8B5M+tcUPhqKunG+JG8pYXqpeYk5WJ7XaqYhILl73oy4C4jgypzUPTCC6wV94EW+sk2gvQ6G6Rtkm3JLQ/f0zvAU7itNsymJnf7Qu4NLTBn8l7JucEAhAJZFC6V7uz6bWVElWcZWNxPnAE3WOAG1C5+jS0qmTjgL48nTcOhIP+RKu1d2ug== Received: from AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) by AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.16; Sun, 26 Sep 2021 06:32:55 +0000 Received: from AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::787b:2156:ca99:fe00]) by AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::787b:2156:ca99:fe00%3]) with mapi id 15.20.4544.021; Sun, 26 Sep 2021 06:32:55 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Sun, 26 Sep 2021 08:32:35 +0200 Message-ID: X-Mailer: git-send-email 2.30.2 X-TMN: [c+1Vjyi9VGKtt1+/XV882hSF3qyVdQog] X-ClientProxiedBy: FR3P281CA0006.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1d::16) To AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) X-Microsoft-Original-Message-ID: <20210926063240.344252-1-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sblaptop.fritz.box (188.192.142.38) by FR3P281CA0006.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.8 via Frontend Transport; Sun, 26 Sep 2021 06:32:54 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4e691f9e-a961-4ccd-0bf0-08d980b7793e X-MS-Exchange-SLBlob-MailProps: 3tYrFWTD79DYpfpsZuOR7dH2ijAywP+xwoyzJCjpTGZa5G4EqxiABoiL2ywBpt5u01hKlm+LzE/hc9d3at9YAl/Y4JoYas+Tw6ZObjmzT4zV7y4LwEpyYAmHZm0S4eO4OGnTv3Lp9oTItNJf15YO6fftMhEVlEKXcAbcMAhIZzRVoVs0VxKAK2D3TkhV4sWKHMjLGJw5xtqmY5maW5uvlao473emXVMISMhWQ3y91P+cyzsIVBLXUnT0ZLPPciEiaT1FOsxba65dEwYknjDRBNBDg5Rqs0lrF+IZZx4jwI1QQPD1IQjwImnzx3xMjjUaT0Xfo4chR7vz3G1l4Ij7mo4bjn8nvvjsZZgE+jU98IwpnzXpW/zltGpVW1E3VDjX25V6KlOrqkPdKDPv4oyx0vtjDk23YGr/YnqXzr/goN/fETlztMw6RGpYB8AEzGo+luuVzukYDQSri+mkejWWmhEan2Hwoh/fNLX2+h5M+dRhV48w0hFf8XiNm7GGq74wUNMtIo5fd2XNSqE0udqkFk+S72I9cqpsqbJIFv6BR6STFnRYR958nHnYyUPXD0LKReHyUh4asU8Z7dKnkTZK022E1NK3JUn8WzQeCfPUKjwI0b/NnBP3mO18U6niDrxU+i9vL7hcFctvry0qjpBaVtQt8mNB0I7j4PtNZ4uMZHRF15FHjfF7G3Xzp+1Qct3PZHB0oJyct2VlCHKQKxaFA+XuczXch8BuA9betEP/DN6SQpwJ7Z50eFuqOPj7vEoi4QOpixOKw3i/fVkx4WilKD+BxC+lf9Uv X-MS-TrafficTypeDiagnostic: AM7PR03MB6660: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hz4mSS8GjrwtFt3ywP5EuLTq1lAvcDy1Csg7vN06ALWCVcJc2ME5nGG7SbQwg7uCjVVVwnFT2ke7j19vb5dlbLn2MXcZ2DE4ndSej4L8xqyb8PXWNz+nqleRPQY2e895mEM8Ukpo+qpiA7M/8K82LcA0QagMEgam+QMEMw19RmMa0KXIbAs1oQiV+bb4mrnh4cZ15nWsAWUrMlGEn+aRHwtjIASpp5hYy5iUHYnJqaQNvl7VuwvX6GSGoPFx7HkUFxyMfMhIvn1RQtiBxPo4DWmGCJYxZewvyXHi96qkX7IF6U3fUTDjO+Gw0BVqtFylJEYG9YQfpSOPJFVMM9m78NQ2rBR/JpG1rPT1xI2yWJTzizdUx94efnfD9fmpzkpql6fJSiLbSrYHGQhtwlq0LvizxsEBPo7ec6KrRSCvOxmjNi0cza6cIIUcR9sXLy2F X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fOTXvVk3CPn6B4GynBZ6mKOZX4vXbxqcyOcWYXNVkcKau+qWGJ2FfP/NRvBTQbNUSVkjCnC4zRtCu6LLyVYRCrU30rIQTltVK/73kzD13zWyFOIKVcRnF2Cbz4IAa67QgjVmXa62mQxsIOP4xrYcPw== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e691f9e-a961-4ccd-0bf0-08d980b7793e X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6660.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Sep 2021 06:32:55.4301 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR03MB6660 Subject: [FFmpeg-devel] [PATCH 1/6] avcodec/qsvenc: Fix leak and crash when encoding H.264 due to A53_CC X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: h3f75tOCuYZS Since commit 3bbe0c210b05fc6fbd7b1d4bbd8479db7f2cf957, the Payloads array of every QSVFrame leaks as soon as the frame is reused; the leak is small and not very noticeable, but if there is an attempt to use said array the ensuing crash is much more noticeable. This happens when encoding H.264 with A53 CC side data. Furthermore, if said array can not be allocated at all, an AVFrame leaks. Fix all of this by not allocating the array separately at all; put it in QSVFrame instead and restore the Payloads array upon reusing the frame. Finally, use av_freep() instead of av_free() to free the payload entries. Signed-off-by: Andreas Rheinhardt --- libavcodec/qsv_internal.h | 2 ++ libavcodec/qsvenc.c | 10 +++------- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h index 8090b748b3..fe9d5319c4 100644 --- a/libavcodec/qsv_internal.h +++ b/libavcodec/qsv_internal.h @@ -76,6 +76,8 @@ typedef struct QSVFrame { mfxExtDecodedFrameInfo dec_info; mfxExtBuffer *ext_param; + mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for enc_ctrl.Payload + int queued; int used; diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index 06f55604b5..66f79bb021 100644 --- a/libavcodec/qsvenc.c +++ b/libavcodec/qsvenc.c @@ -1259,7 +1259,7 @@ static void free_encoder_ctrl_payloads(mfxEncodeCtrl* enc_ctrl) if (enc_ctrl) { int i; for (i = 0; i < enc_ctrl->NumPayload && i < QSV_MAX_ENC_PAYLOAD; i++) { - av_free(enc_ctrl->Payload[i]); + av_freep(&enc_ctrl->Payload[i]); } enc_ctrl->NumPayload = 0; } @@ -1273,6 +1273,7 @@ static void clear_unused_frames(QSVEncContext *q) free_encoder_ctrl_payloads(&cur->enc_ctrl); //do not reuse enc_ctrl from previous frame memset(&cur->enc_ctrl, 0, sizeof(cur->enc_ctrl)); + cur->enc_ctrl.Payload = cur->payloads; if (cur->frame->format == AV_PIX_FMT_QSV) { av_frame_unref(cur->frame); } @@ -1309,11 +1310,7 @@ static int get_free_frame(QSVEncContext *q, QSVFrame **f) av_freep(&frame); return AVERROR(ENOMEM); } - frame->enc_ctrl.Payload = av_mallocz(sizeof(mfxPayload*) * QSV_MAX_ENC_PAYLOAD); - if (!frame->enc_ctrl.Payload) { - av_freep(&frame); - return AVERROR(ENOMEM); - } + frame->enc_ctrl.Payload = frame->payloads; *last = frame; *f = frame; @@ -1615,7 +1612,6 @@ int ff_qsv_enc_close(AVCodecContext *avctx, QSVEncContext *q) while (cur) { q->work_frames = cur->next; av_frame_free(&cur->frame); - av_free(cur->enc_ctrl.Payload); av_freep(&cur); cur = q->work_frames; }