From patchwork Fri Nov 17 17:03:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rogozhkin, Dmitry V" X-Patchwork-Id: 44699 Delivered-To: ffmpegpatchwork2@gmail.com Received: by 2002:a05:6a20:6a89:b0:181:818d:5e7f with SMTP id bi9csp11904pzb; Fri, 17 Nov 2023 09:04:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IGK3r5Y6rL8EhFYko+KHUQHgzndqiLij5ikwYe+mz55E11t2b7Jw31Aj7/xzBnAJf1Vefnk X-Received: by 2002:ac2:43a5:0:b0:501:c406:c296 with SMTP id t5-20020ac243a5000000b00501c406c296mr131434lfl.31.1700240649194; Fri, 17 Nov 2023 09:04:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700240649; cv=none; d=google.com; s=arc-20160816; b=Z3/9ClQSSxAb1JbOtuo19HIK0HR3ZFpxGizn4EbvqiBd1OQF6y5VNNm5j4scCJeGvy xgUZLl6fJuEBqhYDdyhMy5hsypfNE2Bl1gmuKTnToU0FdDAhCbX3d4r6jePU//O6j1jG yFT3iortZEomMet3m986aJ8mKLVUzxF37bfr2SgoxsNjfdatGX8Ywwqzy2D35o+6gqdo tl8rrBUj0K0p4f1QDwSkreYKqI6hLG6ZPRf3iKI+AtEYE2wTPBXwgPkJXrD6z9il47vY xpKrZOVW6E24gmiMFkiosPErmozslsIiy7LV2/PMU2Xfs60i2Cu3ZnvhFj/rRl8sX7t1 9IQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc:reply-to :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=v5wDf/cIRrtEZzvpWwuaShSn66l3ijZMqLBFiTiQxEk=; fh=z34nsZay6HvjYIRPk5BwyArNH+rsauucRUMCMjMxepw=; b=PVI+6EdlrII/rUcWwQub0K+eNX1c/ZW56RPFbUQFuAZzmQlvWuaKwFW0uYGv3gz1KH PVUoGWLgjTZ+6YRlr5dhLxCowscBW8E6gG6Ohz2oHCFLgXi/J2PsMJ1tG0qWc90bdWr0 qyanmJJRky6hJev8lzSvvlaOSUhRE0gBqCqB+rEioXZJQG1Atuuc3VD5xsjvLu0IgPtq ifnxv61e7t13/RG+AZ5rtf7t0B9fVW8bkyf32l2xG+VchLO+HmeOa1uqdY84oQ679+cd eoDwueGFlSSitZzZOjnLlueb2gTZ1W9Qkfs1b57/iBLPb8in0Bw3pGAeQG6cYdAYDZ29 poCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ZT8RvEoI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org. [79.124.17.100]) by mx.google.com with ESMTP id r12-20020aa7c14c000000b00537705d2093si1065251edp.63.2023.11.17.09.03.44; Fri, 17 Nov 2023 09:04:09 -0800 (PST) Received-SPF: pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) client-ip=79.124.17.100; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel.com header.s=Intel header.b=ZT8RvEoI; spf=pass (google.com: domain of ffmpeg-devel-bounces@ffmpeg.org designates 79.124.17.100 as permitted sender) smtp.mailfrom=ffmpeg-devel-bounces@ffmpeg.org Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 3E15B68CD08; Fri, 17 Nov 2023 19:03:39 +0200 (EET) X-Original-To: ffmpeg-devel@ffmpeg.org Delivered-To: ffmpeg-devel@ffmpeg.org Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 44DF468C9A4 for ; Fri, 17 Nov 2023 19:03:32 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1700240617; x=1731776617; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=yaRQMZJq8I7Lp60bkBdUwwQRfiBeoQfw8LSteaRf8JM=; b=ZT8RvEoIv4OHeKlkBw63p2HWN3DdcMs1gC/eYsgwNASCgr0IHfG4He8A wpl9EgdMIJfLT2p/oQqQ4gXff2AA/WV8VlijXQfnZU4tzP6quIo434QkH 6a/s7L7GGcuggOvpuN3RAWyatV0OkkmiZSkVRZKwOtUE738IaBjfDN+4M kZvo6FESmQ/MHq1s6+B6MqtKhxSN0r1jvlq1C6TlHGl1ni1ES1KJKlX2Y hopBQksRfAb0nMjpcIuwEiPidESb97nADCEMZArY+IEN54tviInYr3goJ YrBoHYVNKbG+YLJqyyDWjEVxDu2jsCdniIE7ZsSS+W+BvfLG6oUUJMXVW w==; X-IronPort-AV: E=McAfee;i="6600,9927,10897"; a="394181420" X-IronPort-AV: E=Sophos;i="6.04,206,1695711600"; d="scan'208";a="394181420" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Nov 2023 09:03:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10897"; a="939199740" X-IronPort-AV: E=Sophos;i="6.04,206,1695711600"; d="scan'208";a="939199740" Received: from dvrscl.jf.intel.com ([10.54.72.26]) by orsmga005.jf.intel.com with ESMTP; 17 Nov 2023 09:03:29 -0800 From: Dmitry Rogozhkin To: ffmpeg-devel@ffmpeg.org Date: Fri, 17 Nov 2023 09:03:03 -0800 Message-Id: <1700240583-793-1-git-send-email-dmitry.v.rogozhkin@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1699986151-24741-1-git-send-email-dmitry.v.rogozhkin@intel.com> References: <1699986151-24741-1-git-send-email-dmitry.v.rogozhkin@intel.com> Subject: [FFmpeg-devel] [PATCH v3] avcodec/decode: guard against NULL hw_frames_ctx X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Christoph Reiter , Dmitry Rogozhkin , Lynne MIME-Version: 1.0 Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" X-TUID: 0ie4g8eQpePk Guard against segfault running VLC decoding under msys2 [1]: Thread 33 received signal SIGSEGV, Segmentation fault. [Switching to Thread 37728.0xadd0] ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) at libavcodec/decode.c:1848 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; (gdb) bt at libavcodec/decode.c:1848 at libavcodec/h264_slice.c:208 first_slice=1) at libavcodec/h264_slice.c:1599 at libavcodec/h264_slice.c:2130 at libavcodec/h264dec.c:652 got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 (gdb) p avctx $1 = (AVCodecContext *) 0x6447b00 (gdb) p avctx->hw_frames_ctx $2 = (AVBufferRef *) 0x0 v2: check for free_frame_priv (Hendrik) v3: return EINVAL (Christoph Reiter) See[1]: https://github.com/msys2/MINGW-packages/pull/19050 Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") CC: Lynne CC: Christoph Reiter Signed-off-by: Dmitry Rogozhkin --- libavcodec/decode.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/libavcodec/decode.c b/libavcodec/decode.c index ad39021..50c3995 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_private) { const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); - AVHWFramesContext *frames_ctx; if (!hwaccel || !hwaccel->frame_priv_data_size) return 0; av_assert0(!*hwaccel_picture_private); - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, - frames_ctx->device_ctx, - hwaccel->free_frame_priv); + if (hwaccel->free_frame_priv) { + AVHWFramesContext *frames_ctx; + + if (!avctx->hw_frames_ctx) + return AVERROR(EINVAL); + + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, + frames_ctx->device_ctx, + hwaccel->free_frame_priv); + } else { + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel->frame_priv_data_size); + } + if (!*hwaccel_picture_private) return AVERROR(ENOMEM);