From patchwork Tue Nov 14 19:14:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rogozhkin, Dmitry V" X-Patchwork-Id: 44666 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:92a5:b0:181:818d:5e7f with SMTP id q37csp2206025pzg; Tue, 14 Nov 2023 11:15:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEQBNesOkzhXCXZlJkhPP8tRidJfUqwFkqv5M0GsyQWJtw7FCG633ffJM4EzCo02XV0Ybj X-Received: by 2002:a17:906:118e:b0:9ee:a767:12e7 with SMTP id n14-20020a170906118e00b009eea76712e7mr2662404eja.6.1699989357996; Tue, 14 Nov 2023 11:15:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699989357; cv=none; d=google.com; s=arc-20160816; b=Q2xQT+K+oxW2bO3ZSz0lxLjdQW1Zx9WBi0npUAqtbL2MUF9IFEKzPC9tWfGWt71tWD 7usXu95BsIJpYiPV4vQpz0f0ZyRphiRZyvU1eNfTEKMdzxMu+De9P9HV0kbpW9865Sm4 MxCCwv0EFfu+UEOC986Ph4ZoL3wsvxY0WO6KNqLxvehHRXelagbKutUCC+kz00aS3vd5 zf1SSBazoqeezLUxqb2EtmQqG8j5mvFNfsZ3fNOvHUivCrGBWYilgQDZFftDf2Pk+JSu OTx73sXVzHLGaRoWV6FxlMYykM2GB9DOybPyzMvMzLPcRdaHHAucWqpkR102408+cVG8 Vhnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=Y4lfBWZuraJYLVFsh3YBy6v4Q/xadIyEIekLFPGiuso=; fh=z34nsZay6HvjYIRPk5BwyArNH+rsauucRUMCMjMxepw=; b=NmGkgmmhUx0/cTt8xJ7Jo2Te5WKx/HWhLdkgIpbc8AshyxiwCIrYT/ugzdi9w1LCw9 0Si3xFCWLlk8kDBMuBvRpVsmYnQiu63kWeT2/yZ7PjawY6KVHLjEHMcK4I3PZwKieOjm QGkr37cxggAa2lpdMrBscy8qiKzV4v4Hp/ANZad6uKsa+DJq7sNDApHXf2RtE9ID5ySM LZJLGV0/zfbFWsMDdhiaepzkP2Jx2Ea47s9LGdG6i1Ox6fek1e+W8ubaEffiMcWsDN3k SLYd9FGe3cTzUtYnnqzRVAoWzrocc2CWvP5KWl8Y/cz5ItF/B69rEA6XB/yi/sHLzCM5 AOWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ZlrTmpo6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id ji7-20020a170907980700b009e6bf7b9b3asi4266983ejc.866.2023.11.14.11.15.38; Tue, 14 Nov 2023 11:15:57 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ZlrTmpo6; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3E35868CD55; Tue, 14 Nov 2023 21:15:36 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.65]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id AEEFB68CC2E for ; Tue, 14 Nov 2023 21:15:28 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1699989334; x=1731525334; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=YFMa0JhB1miIlqBGc0KjVHKvxJlBuXGXZr2aahsERcM=; b=ZlrTmpo6jS/Jh/N1Noq3Mknc57LMPOc+Lvm7rPEBXoxKUtz/Cn5K3k5O oio3e7E3kS0+60Y64sD4Qy7xJwdE3dgqXYjI4ig0LQzoVjPRVDWvbdlKt PnBjOib9io7WbftGtaTQv0ofXi4YBsZw+zGRu7xXdb6c5E537106kSzD4 0q+Kur4n5PZC9WfQjreJWuphBuHTPY10175d7FqjoR4kJNcBoZ3M01gMz 3h1Ze+DDYHkMOklqKoBtC09ATB2TED0/Z4n5FdZQtqOKNopgyyCmNb59e L/kgUJjHgEofo1hT1hW2yrqQrEy6mZsam4Czbk/QxRJFMKPQr2CLKD3wc A==; X-IronPort-AV: E=McAfee;i="6600,9927,10894"; a="394645284" X-IronPort-AV: E=Sophos;i="6.03,302,1694761200"; d="scan'208";a="394645284" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Nov 2023 11:15:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10894"; a="714654180" X-IronPort-AV: E=Sophos;i="6.03,302,1694761200"; d="scan'208";a="714654180" Received: from dvrscl.jf.intel.com ([10.54.72.26]) by orsmga003.jf.intel.com with ESMTP; 14 Nov 2023 11:15:12 -0800 From: Dmitry Rogozhkin To: ffmpeg-devel@ffmpeg.org Date: Tue, 14 Nov 2023 11:14:32 -0800 Message-Id: <1699989272-28075-1-git-send-email-dmitry.v.rogozhkin@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1699986151-24741-1-git-send-email-dmitry.v.rogozhkin@intel.com> References: <1699986151-24741-1-git-send-email-dmitry.v.rogozhkin@intel.com> Subject: [FFmpeg-devel] [PATCH v2] avcodec/decode: guard against NULL hw_frames_ctx X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Christoph Reiter , Dmitry Rogozhkin , Lynne MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: vfw53eO2uubc Guard against segfault running VLC decoding under msys2 [1]: Thread 33 received signal SIGSEGV, Segmentation fault. [Switching to Thread 37728.0xadd0] ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) at libavcodec/decode.c:1848 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; (gdb) bt at libavcodec/decode.c:1848 at libavcodec/h264_slice.c:208 first_slice=1) at libavcodec/h264_slice.c:1599 at libavcodec/h264_slice.c:2130 at libavcodec/h264dec.c:652 got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 (gdb) p avctx $1 = (AVCodecContext *) 0x6447b00 (gdb) p avctx->hw_frames_ctx $2 = (AVBufferRef *) 0x0 v2: check for free_frame_priv (Hendrik) See[1]: https://github.com/msys2/MINGW-packages/pull/19050 Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") CC: Lynne CC: Christoph Reiter Signed-off-by: Dmitry Rogozhkin --- libavcodec/decode.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/libavcodec/decode.c b/libavcodec/decode.c index ad39021..58f887d 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1838,17 +1838,29 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_private) { const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); - AVHWFramesContext *frames_ctx; if (!hwaccel || !hwaccel->frame_priv_data_size) return 0; av_assert0(!*hwaccel_picture_private); - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, - frames_ctx->device_ctx, - hwaccel->free_frame_priv); + if (hwaccel->free_frame_priv) { + AVHWFramesContext *frames_ctx; + + if (!avctx->hw_frames_ctx) + return AVERROR(ENOMEM); + + frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx->data; + if (!frames_ctx) + return AVERROR(ENOMEM); + + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, + frames_ctx->device_ctx, + hwaccel->free_frame_priv); + } else { + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel->frame_priv_data_size); + } + if (!*hwaccel_picture_private) return AVERROR(ENOMEM);