| Message ID | 1700240583-793-1-git-send-email-dmitry.v.rogozhkin@intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel,v3] avcodec/decode: guard against NULL hw_frames_ctx | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/make_loongarch64 | success | Make finished |
| yinshiyou/make_fate_loongarch64 | success | Make fate finished |
| andriy/make_x86 | success | Make finished |
| andriy/make_fate_x86 | success | Make fate finished |
On 2023-11-17 09:03:03 -0800, Dmitry Rogozhkin wrote: > Guard against segfault running VLC decoding under msys2 [1]: > > Thread 33 received signal SIGSEGV, Segmentation fault. > [Switching to Thread 37728.0xadd0] > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) > at libavcodec/decode.c:1848 > 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > (gdb) bt > at libavcodec/decode.c:1848 > at libavcodec/h264_slice.c:208 > first_slice=1) at libavcodec/h264_slice.c:1599 > at libavcodec/h264_slice.c:2130 > at libavcodec/h264dec.c:652 > got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 > > (gdb) p avctx > $1 = (AVCodecContext *) 0x6447b00 > (gdb) p avctx->hw_frames_ctx > $2 = (AVBufferRef *) 0x0 > > v2: check for free_frame_priv (Hendrik) > v3: return EINVAL (Christoph Reiter) > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") > CC: Lynne <dev@lynne.ee> > CC: Christoph Reiter <reiter.christoph@gmail.com> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> We have also seen the same issue on Debian unstable with VDPAU playback in vlc. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055952. I can confirm that this patch fixes the issue. Please apply the patch and backport it to 6.1. Cheers > --- > libavcodec/decode.c | 18 +++++++++++++----- > 1 file changed, 13 insertions(+), 5 deletions(-) > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c > index ad39021..50c3995 100644 > --- a/libavcodec/decode.c > +++ b/libavcodec/decode.c > @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) > int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_private) > { > const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); > - AVHWFramesContext *frames_ctx; > > if (!hwaccel || !hwaccel->frame_priv_data_size) > return 0; > > av_assert0(!*hwaccel_picture_private); > > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, > - frames_ctx->device_ctx, > - hwaccel->free_frame_priv); > + if (hwaccel->free_frame_priv) { > + AVHWFramesContext *frames_ctx; > + > + if (!avctx->hw_frames_ctx) > + return AVERROR(EINVAL); > + > + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, > + frames_ctx->device_ctx, > + hwaccel->free_frame_priv); > + } else { > + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel->frame_priv_data_size); > + } > + > if (!*hwaccel_picture_private) > return AVERROR(ENOMEM); > > -- > 1.8.3.1 > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Fri, Nov 17, 2023 at 6:04 PM Dmitry Rogozhkin <dmitry.v.rogozhkin-at-intel.com@ffmpeg.org> wrote: > > Guard against segfault running VLC decoding under msys2 [1]: > > Thread 33 received signal SIGSEGV, Segmentation fault. > [Switching to Thread 37728.0xadd0] > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) > at libavcodec/decode.c:1848 > 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > (gdb) bt > at libavcodec/decode.c:1848 > at libavcodec/h264_slice.c:208 > first_slice=1) at libavcodec/h264_slice.c:1599 > at libavcodec/h264_slice.c:2130 > at libavcodec/h264dec.c:652 > got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 > > (gdb) p avctx > $1 = (AVCodecContext *) 0x6447b00 > (gdb) p avctx->hw_frames_ctx > $2 = (AVBufferRef *) 0x0 > > v2: check for free_frame_priv (Hendrik) > v3: return EINVAL (Christoph Reiter) > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") > CC: Lynne <dev@lynne.ee> > CC: Christoph Reiter <reiter.christoph@gmail.com> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > --- The latest change itself looks fine to me, I would however prefer if the commit message would contain a bit more text and a bit less gdb dump. Maybe something like this (quick suggestion, for title and body): avcodec: validate hw_frames_ctx is available when AVHWAccel.free_frame_priv is used Validate that a hw_frames_ctx is available before using it for the AVHWAccel.free_frame_priv callback, and don't require it to be present when the callback is not in use by the HWAccel. --- Feel free to augment and reword, but I don't think the gdb dump offers much info that couldn't be conveyed more clearly in a few words that mention the absence of hw_frame_ctx. - Hendrik
On Sun, 2023-11-19 at 16:29 +0100, Hendrik Leppkes wrote: > On Fri, Nov 17, 2023 at 6:04 PM Dmitry Rogozhkin > <dmitry.v.rogozhkin-at-intel.com@ffmpeg.org> wrote: > > > > Guard against segfault running VLC decoding under msys2 [1]: > > > > Thread 33 received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 37728.0xadd0] > > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, > > hwaccel_picture_private=0x65dfd00) > > at libavcodec/decode.c:1848 > > 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx- > > >data; > > (gdb) bt > > at libavcodec/decode.c:1848 > > at libavcodec/h264_slice.c:208 > > first_slice=1) at libavcodec/h264_slice.c:1599 > > at libavcodec/h264_slice.c:2130 > > at libavcodec/h264dec.c:652 > > got_frame=0x646e4b0, avpkt=0x64522c0) at > > libavcodec/h264dec.c:1048 > > > > (gdb) p avctx > > $1 = (AVCodecContext *) 0x6447b00 > > (gdb) p avctx->hw_frames_ctx > > $2 = (AVBufferRef *) 0x0 > > > > v2: check for free_frame_priv (Hendrik) > > v3: return EINVAL (Christoph Reiter) > > > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv > > callback") > > CC: Lynne <dev@lynne.ee> > > CC: Christoph Reiter <reiter.christoph@gmail.com> > > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > > --- > > The latest change itself looks fine to me, I would however prefer if > the commit message would contain a bit more text and a bit less gdb > dump. > > Maybe something like this (quick suggestion, for title and body): > > avcodec: validate hw_frames_ctx is available when > AVHWAccel.free_frame_priv is used > > Validate that a hw_frames_ctx is available before using it for the > AVHWAccel.free_frame_priv callback, and don't require it to be > present when > the callback is not in use by the HWAccel. I am fine with that. Thank you for the text. Added with minor change in title to shorten it. See v4 patch. > > --- > > Feel free to augment and reword, but I don't think the gdb dump > offers > much info that couldn't be conveyed more clearly in a few words that > mention the absence of hw_frame_ctx. > > - Hendrik > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
diff --git a/libavcodec/decode.c b/libavcodec/decode.c index ad39021..50c3995 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_private) { const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); - AVHWFramesContext *frames_ctx; if (!hwaccel || !hwaccel->frame_priv_data_size) return 0; av_assert0(!*hwaccel_picture_private); - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, - frames_ctx->device_ctx, - hwaccel->free_frame_priv); + if (hwaccel->free_frame_priv) { + AVHWFramesContext *frames_ctx; + + if (!avctx->hw_frames_ctx) + return AVERROR(EINVAL); + + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, + frames_ctx->device_ctx, + hwaccel->free_frame_priv); + } else { + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel->frame_priv_data_size); + } + if (!*hwaccel_picture_private) return AVERROR(ENOMEM);
Guard against segfault running VLC decoding under msys2 [1]: Thread 33 received signal SIGSEGV, Segmentation fault. [Switching to Thread 37728.0xadd0] ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) at libavcodec/decode.c:1848 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; (gdb) bt at libavcodec/decode.c:1848 at libavcodec/h264_slice.c:208 first_slice=1) at libavcodec/h264_slice.c:1599 at libavcodec/h264_slice.c:2130 at libavcodec/h264dec.c:652 got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 (gdb) p avctx $1 = (AVCodecContext *) 0x6447b00 (gdb) p avctx->hw_frames_ctx $2 = (AVBufferRef *) 0x0 v2: check for free_frame_priv (Hendrik) v3: return EINVAL (Christoph Reiter) See[1]: https://github.com/msys2/MINGW-packages/pull/19050 Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") CC: Lynne <dev@lynne.ee> CC: Christoph Reiter <reiter.christoph@gmail.com> Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> --- libavcodec/decode.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)