From patchwork Mon Nov 29 14:36:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Rheinhardt X-Patchwork-Id: 31793 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a6b:d206:0:0:0:0:0 with SMTP id q6csp6231066iob; Mon, 29 Nov 2021 06:37:02 -0800 (PST) X-Google-Smtp-Source: ABdhPJz+hYuOdJrWi46ytvxklWHlZ0+dmkymJukTPWahR8vQPH6JwQkSg9fmGdYF7BShGCh4W3jF X-Received: by 2002:a05:6402:d49:: with SMTP id ec9mr74060776edb.235.1638196622130; Mon, 29 Nov 2021 06:37:02 -0800 (PST) Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id u9si21649178edr.131.2021.11.29.06.37.00; Mon, 29 Nov 2021 06:37:02 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@outlook.com header.s=selector1 header.b=e92VWzEt; arc=fail (body hash mismatch); spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=outlook.com Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id B4F0968A905; Mon, 29 Nov 2021 16:36:55 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-oln040092074028.outbound.protection.outlook.com [40.92.74.28]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id ABEAD68807F for ; Mon, 29 Nov 2021 16:36:48 +0200 (EET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZVaoS08DcxO9HQyfvKP3s1czAamTsJjXPos96edlPf3H6jqYKkxuQyVitEEw0+voQbKNyft5vF3t5Ehzqzspnbw1JTcu2QRoiYJ0ahvXLjK7YT8x/56U2VhvQY+xr4yi9LEU+b2+kyjqS0QNwkRdLunUqJFZNStGX01vseuoJP4DajTYKI13zTsQSoCc8MaHQ6T9GlYW0hA9f/yXrnPQgRthpYIGHksJ9/bylmX9fQCvDVm65S7880d8C1ZTH+YfF1vHB68MVHcz85hB5f0epYeSfkqpWWg0F+5OhfeuC/Q9S9np4CzW0RFOM19GIsc/C0dw2PXpsSHZu/fQRBIX+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gg++gh9XsPjfBFhzR6bxHypCXkMpZd7pFLOJOVZYn+I=; b=mDZRXwn/t7vRvDi+46JllG2lSLoNzJb/2cECdu/460wzBuvUHBOSaLhjYOGdeInXLhuFczosuzygOdcY5rJFkFzw29dmJ/d3VQPZSequOcMCLRpPZ9VmOhD4hqPAAceZZGW/TyamaabX/cqQ+o+L4Ptaax34Q/LXJ/zaLtoudKvChpTf4DBr+zPM0JZEFYI/6Cf8hattUTDVZM5CXf0Ia8fsuVpS+La17aRSmTCLMPztPojYpsKySm5GAHDqM/9/S/RbF0ovy8QP3pPRMarzyiWzEtNKu832SiO5q8XH4LwmMK4dgAF4b/4rkPjjmDL3JDDS4m7bIEN3Ky/WlxFTvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gg++gh9XsPjfBFhzR6bxHypCXkMpZd7pFLOJOVZYn+I=; b=e92VWzEt3zNej5MYwynP9Z6v5gWI2Rm61BtPAEB1TCA7BhLPdsz5pr/ZLHweK0oiFRZOoPKuHV3J2Uq9RsKr7o3vc7nz4nE3jIP8iEzuIQhIrUMuxi1MKnKicpeNgThd2nGza5OhLk/RXSAxvdY+XgRklX3WlWxUviSvdmCjk/JcEII3P7z7wI+rjuoJP3FKTkniTHWSHRmpGS+H9sTi99lhzT5auZUEj5fIwyjqCT2Tw/YuUtGLg3HggfNbFWEawEyJ0PSddr2bbXu54+aBqB2qnG8aX0Fv9Zqh6ZduVh9MJsaVJ3/ksTYhSTxadc1svWXtDxxFbwZML+f77CgNzw== Received: from AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) by AM6PR03MB5703.eurprd03.prod.outlook.com (2603:10a6:20b:fb::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Mon, 29 Nov 2021 14:36:45 +0000 Received: from AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::700f:d70b:3bb8:4d51]) by AM7PR03MB6660.eurprd03.prod.outlook.com ([fe80::700f:d70b:3bb8:4d51%9]) with mapi id 15.20.4734.024; Mon, 29 Nov 2021 14:36:45 +0000 From: Andreas Rheinhardt To: ffmpeg-devel@ffmpeg.org Date: Mon, 29 Nov 2021 15:36:36 +0100 Message-ID: X-Mailer: git-send-email 2.32.0 X-TMN: [aSz9SUJ855obY4VXQYECTu2rt26CIK+L] X-ClientProxiedBy: AM0PR01CA0160.eurprd01.prod.exchangelabs.com (2603:10a6:208:aa::29) To AM7PR03MB6660.eurprd03.prod.outlook.com (2603:10a6:20b:1c1::22) X-Microsoft-Original-Message-ID: <20211129143637.21731-1-andreas.rheinhardt@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from sblaptop.fritz.box (188.192.248.74) by AM0PR01CA0160.eurprd01.prod.exchangelabs.com (2603:10a6:208:aa::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.20 via Frontend Transport; Mon, 29 Nov 2021 14:36:44 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2ab38f8f-64f0-4149-67f5-08d9b345aaa9 X-MS-Exchange-SLBlob-MailProps: S/btQ8cKWiRpLJNj1ZiGHut8HTRQiEaGHsaPGDnKn3luBkpMnKvmbtPNUMh0kztPbCxkSRSNqbAmIz8yLK1zzskZORSP+Ezo3uhZqKaiSf9jKzWiJoYHI4ECd5n5+5Q9TX05m2GI/yQUI6UJwRUObqocc/kodTjVVzJTFFXdKAtN0iDmhR9T1GWhvUJcyfR/IRv2qUhU05bIbqYS633plyCKHO64ce8zLJ6Dq+J7+fN8TCLHe7nbeEffeKMCk9dBTe3FrdaxO3Wl/HJDJKomBw8MXX071FrsFEvyaFI27Fmp2mYRPQ31I6GMmK4t1blkq91xhQR679zM0UYd9txnYmMS1ARZK0hOVLb/uoT8IlS3bIDJwwiVF0gMgVLKNOjirj3iqQWOzV1X+ddBYbKOeLWcLY+wqv4Clr2u4opa5ALGYYVC/poOEQQ3GZhSf1qdPPgdoWk2zZkUW25Toe6sKZ708H6OvCziQFoPjULlGYNR5NkwyU20q7QM8MCyv3hl5dHcCWDrXpsyyjSK1pviPGFT6j+JQZcew+Utx2SztE0/ydQz3L6J0DXsC1TxsVbWzI8DPxxjOEp7tuUfJzg//qopbn9Xb1ml5bJbVMzzU7nazjUCjCtocH7xfR9k9z/tXadJnyGygxUF/I6W8qX7YsMQPb+N9gQ3nG5oJBE/+wOmksyQ2FeX7LqWxGNsWR+nQ61VILEkbVLHnuu75scLI8qVGzuOBAkDdnj8VPusYrc92bpioQBbnmpznE4Mtj1HNhy7Qiad65g= X-MS-TrafficTypeDiagnostic: AM6PR03MB5703: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uqaqU6ds06Gyj+smD6oL6WRrUf+BheImJAEmYwnWoHbn1kYgOM+f4PEzdWjrWi6K+VKtQOt2P4+2DIG0j1Hszjnjxnjx7HlQbEsCcV6eLnbrDYbcSyWSdY4MlNB99Dr89EWIolR2VsYavbjvI0pc3856Z8YdSys0xUGPQAp9kbrk11Jzh09O+WZPmIT+rxBg/THR/rH+nD1guwZcCxdVSieT8cr5cga/REqV36bLj1jcycUeY7D8xNcExmk1z975vxEPRCsQi5PZMl5TfD9uH76OZ9Pd3GHLoEV5+GPm3LHZ8RXopJdSPmRTquMeWtr77dhdlNAQY5aTHV/GqHVV2mQwLjgPbhCoBgUBL/oWQhdY5IErnpnKAsFbvFepnia7FHwQnSeqf0Iu3Zhl67tZByuxAqVEZZGWGdr0VLiI8ltZCpNxyry93/XaZ8qSQZf7QNlStzE1gzv+DXI1ddIpEsxWAt0AIVc1uOyW+EE/wXaSnc6Z/Ia/rUjx4uZPPipb+Dk3Zaxf8wTC6sDkrSTpjRNbO7QCzAS1xBNtyojo7n60kOOG6uno5lh+il8Q9T7J77SOdNNOlhIUlBW6hkXiZQ== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8bIRrlvl14Oaf1/gq22V/yCTQJ5q5ZxBgaQqd4yDY6QqssYbjab2pUgB72yS8b9rrDi9+caak+AO711rqkz7q3oikZmFPJ6rg1QjHsg0OwWDsY95QmQ2SYEM59Srz/y3Zb2/Z2WvGfVl74nVIW+JDWPSqy6D/mZT9xMxnlBL/sqvzTyRtfPwVfDyQ45mewV8KCuoxXEsbCEYA/VdQ5KF46IK/26Yyfiq8cpWDjt1BafqOqRnseNch6Vrk3kLcYb/Kd89mOYovWuKcZpNyrOieKK8OG5Vf8FCs7Vqz7MG8GE2mrhE7vFIZAwJeGgKB+k0Qkvnc5VAIwfpAzx4TaT2fzEUGMdFRN75o9bBrlOgKfqlQ58p8SEKpoDkqf+be/DjM0qZsF4gpBPWLh/28ErH3opsw0H5yCz+s45bXY+/a+isFqvzt09D842N+VQ25sMb1ZNeV2FA/UDlfp51WNtNlh/5xXKDvzVgSFPG/UdlW+atuGH30E6BzZ6b+7K/wG4VSpDIm/znZGw85lNH8MvSZHmTZ2/y34CEOrkrLopTbO2p4vAYwjiXrXP2qTW15gsdlfU0YIEhOwo42fgGkKJBXSCxzme9kPYtJvA9LKbWZjVylkZmkRej0q0GHAeBGipBRlGyP67m+x0j1SqapFYnRp5XZbDhAhE8GZPNv+hGyEa6XA7LkfbQsCzEjvRxIoswbQT+teVpZrmL/YEZvkM8TL7L42PHzDfZlh7na7efgTKobQv4xjQeS1tiHaY1KeA31MyuBPxU9VjyjYsTzXX2C6a17T8IgCMbaZN5ZoQYNUnqPslceYa/9DFbVYm63zkOlHOI0QQ7plnTgIdkbNEhOPPD1/76DdAC8fy/75a1/Vb4nLWS97BkK7I5QOStbv0/fdpNJ9DCky48YUj7FNkI7m1pvVYBJIHbaFlyRyB6GVOz7/Nfi8IzryrlTSOwu0ZnrpzY20C5IZsFxHwGmfe4PA== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2ab38f8f-64f0-4149-67f5-08d9b345aaa9 X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6660.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2021 14:36:45.1062 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR03MB5703 Subject: [FFmpeg-devel] [PATCH] avcodec/vaapi_encode: Fix segfault upon closing uninitialized encoder X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Andreas Rheinhardt Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 74N+M2RJVJVe Fixes ticket #9537. Probably a regression since 2b3206891649f317c20993411efef4bee39ae784. Signed-off-by: Andreas Rheinhardt --- libavcodec/vaapi_encode.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c index ec054ae701..3bf379b1a0 100644 --- a/libavcodec/vaapi_encode.c +++ b/libavcodec/vaapi_encode.c @@ -2366,6 +2366,11 @@ av_cold int ff_vaapi_encode_init(AVCodecContext *avctx) VAStatus vas; int err; + ctx->va_config = VA_INVALID_ID; + ctx->va_context = VA_INVALID_ID; + + /* If you add something that can fail above this av_frame_alloc(), + * modify ff_vaapi_encode_close() accordingly. */ ctx->frame = av_frame_alloc(); if (!ctx->frame) { return AVERROR(ENOMEM); @@ -2377,9 +2382,6 @@ av_cold int ff_vaapi_encode_init(AVCodecContext *avctx) return AVERROR(EINVAL); } - ctx->va_config = VA_INVALID_ID; - ctx->va_context = VA_INVALID_ID; - ctx->input_frames_ref = av_buffer_ref(avctx->hw_frames_ctx); if (!ctx->input_frames_ref) { err = AVERROR(ENOMEM); @@ -2531,6 +2533,11 @@ av_cold int ff_vaapi_encode_close(AVCodecContext *avctx) VAAPIEncodeContext *ctx = avctx->priv_data; VAAPIEncodePicture *pic, *next; + /* We check ctx->frame to know whether ff_vaapi_encode_init() + * has been called and va_config/va_context initialized. */ + if (!ctx->frame) + return 0; + for (pic = ctx->pic_start; pic; pic = next) { next = pic->next; vaapi_encode_free(avctx, pic);