| Message ID | 1699986151-24741-1-git-send-email-dmitry.v.rogozhkin@intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] avcodec/decode: guard against NULL hw_frames_ctx | expand |
| Context | Check | Description |
|---|---|---|
| yinshiyou/make_fate_loongarch64 | success | Make fate finished |
| yinshiyou/make_loongarch64 | warning | New warnings during build |
| andriy/make_fate_x86 | success | Make fate finished |
| andriy/make_x86 | warning | New warnings during build |
On Tue, Nov 14, 2023 at 7:23 PM Dmitry Rogozhkin <dmitry.v.rogozhkin-at-intel.com@ffmpeg.org> wrote: > > Gurd against segfault running VLC decoding under msys2 [1]: > > Thread 33 received signal SIGSEGV, Segmentation fault. > [Switching to Thread 37728.0xadd0] > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) > at libavcodec/decode.c:1848 > 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > (gdb) bt > at libavcodec/decode.c:1848 > at libavcodec/h264_slice.c:208 > first_slice=1) at libavcodec/h264_slice.c:1599 > at libavcodec/h264_slice.c:2130 > at libavcodec/h264dec.c:652 > got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 > > (gdb) p avctx > $1 = (AVCodecContext *) 0x6447b00 > (gdb) p avctx->hw_frames_ctx > $2 = (AVBufferRef *) 0x0 > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") > CC: Lynne <dev@lynne.ee> > CC: Christoph Reiter <reiter.christoph@gmail.com> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > --- > libavcodec/decode.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c > index ad39021..3caaeec 100644 > --- a/libavcodec/decode.c > +++ b/libavcodec/decode.c > @@ -1845,9 +1845,9 @@ int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_pr > > av_assert0(!*hwaccel_picture_private); > > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > + frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx ? avctx->hw_frames_ctx->data : NULL; > *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, > - frames_ctx->device_ctx, > + frames_ctx ? frames_ctx->device_ctx : NULL, > hwaccel->free_frame_priv); The free_frame_priv callback is not usable when no context is available. The code should be updated to check if the hwaccel has a free_frame_priv callback and then require a frames_ctx to be set and otherwise error out (instead of crashing), and if free_frame_priv is not set, then it can allocate a buffer without these requirements. - Hendrik
On Tue, 2023-11-14 at 19:32 +0100, Hendrik Leppkes wrote: > On Tue, Nov 14, 2023 at 7:23 PM Dmitry Rogozhkin > <dmitry.v.rogozhkin-at-intel.com@ffmpeg.org> wrote: > > > > Gurd against segfault running VLC decoding under msys2 [1]: > > > > Thread 33 received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 37728.0xadd0] > > ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, > > hwaccel_picture_private=0x65dfd00) > > at libavcodec/decode.c:1848 > > 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx- > > >data; > > (gdb) bt > > at libavcodec/decode.c:1848 > > at libavcodec/h264_slice.c:208 > > first_slice=1) at libavcodec/h264_slice.c:1599 > > at libavcodec/h264_slice.c:2130 > > at libavcodec/h264dec.c:652 > > got_frame=0x646e4b0, avpkt=0x64522c0) at > > libavcodec/h264dec.c:1048 > > > > (gdb) p avctx > > $1 = (AVCodecContext *) 0x6447b00 > > (gdb) p avctx->hw_frames_ctx > > $2 = (AVBufferRef *) 0x0 > > > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv > > callback") > > CC: Lynne <dev@lynne.ee> > > CC: Christoph Reiter <reiter.christoph@gmail.com> > > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > > --- > > libavcodec/decode.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c > > index ad39021..3caaeec 100644 > > --- a/libavcodec/decode.c > > +++ b/libavcodec/decode.c > > @@ -1845,9 +1845,9 @@ int > > ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void > > **hwaccel_picture_pr > > > > av_assert0(!*hwaccel_picture_private); > > > > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > > + frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx ? > > avctx->hw_frames_ctx->data : NULL; > > *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel- > > >frame_priv_data_size, 0, > > - frames_ctx- > > >device_ctx, > > + frames_ctx ? > > frames_ctx->device_ctx : NULL, > > hwaccel- > > >free_frame_priv); > > The free_frame_priv callback is not usable when no context is > available. The code should be updated to check if the hwaccel has a > free_frame_priv callback and then require a frames_ctx to be set and > otherwise error out (instead of crashing), and if free_frame_priv is > not set, then it can allocate a buffer without these requirements. Thank you for feedback. I hope I have fixed this and sent v2 update yesterday. Can you, please, help review again? > > - Hendrik > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Nov 21, 2023, 05:38 by dmitry.v.rogozhkin-at-intel.com@ffmpeg.org: > Validate that a hw_frames_ctx is available before using it for > the AVHWAccel.free_frame_priv callback, and don't require it to > be present when the callback is not in use by the HWAccel. > > v2: check for free_frame_priv (Hendrik) > v3: return EINVAL (Christoph Reiter) > v4: better commit message (Hendrik) > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") > CC: Lynne <dev@lynne.ee> > CC: Christoph Reiter <reiter.christoph@gmail.com> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > --- > libavcodec/decode.c | 18 +++++++++++++----- > 1 file changed, 13 insertions(+), 5 deletions(-) > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c > index ad39021..50c3995 100644 > --- a/libavcodec/decode.c > +++ b/libavcodec/decode.c > @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const AVPacket *src, void *logctx) > int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_private) > { > const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); > - AVHWFramesContext *frames_ctx; > > if (!hwaccel || !hwaccel->frame_priv_data_size) > return 0; > > av_assert0(!*hwaccel_picture_private); > > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, > - frames_ctx->device_ctx, > - hwaccel->free_frame_priv); > + if (hwaccel->free_frame_priv) { > + AVHWFramesContext *frames_ctx; > + > + if (!avctx->hw_frames_ctx) > + return AVERROR(EINVAL); > + > + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, > + frames_ctx->device_ctx, > + hwaccel->free_frame_priv); > + } else { > + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel->frame_priv_data_size); > + } > + > if (!*hwaccel_picture_private) > return AVERROR(ENOMEM); > You never set frames_ctx in this patch.
On Tue, 2023-11-21 at 06:17 +0100, Lynne wrote: > Nov 21, 2023, 05:38 by dmitry.v.rogozhkin-at-intel.com@ffmpeg.org: > > > Validate that a hw_frames_ctx is available before using it for > > the AVHWAccel.free_frame_priv callback, and don't require it to > > be present when the callback is not in use by the HWAccel. > > > > v2: check for free_frame_priv (Hendrik) > > v3: return EINVAL (Christoph Reiter) > > v4: better commit message (Hendrik) > > > > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 > > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv > > callback") > > CC: Lynne <dev@lynne.ee> > > CC: Christoph Reiter <reiter.christoph@gmail.com> > > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> > > --- > > libavcodec/decode.c | 18 +++++++++++++----- > > 1 file changed, 13 insertions(+), 5 deletions(-) > > > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c > > index ad39021..50c3995 100644 > > --- a/libavcodec/decode.c > > +++ b/libavcodec/decode.c > > @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const > > AVPacket *src, void *logctx) > > int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void > > **hwaccel_picture_private) > > { > > const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); > > - AVHWFramesContext *frames_ctx; > > > > if (!hwaccel || !hwaccel->frame_priv_data_size) > > return 0; > > > > av_assert0(!*hwaccel_picture_private); > > > > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; > > - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel- > > >frame_priv_data_size, 0, > > - frames_ctx- > > >device_ctx, > > - hwaccel- > > >free_frame_priv); > > + if (hwaccel->free_frame_priv) { > > + AVHWFramesContext *frames_ctx; > > + > > + if (!avctx->hw_frames_ctx) > > + return AVERROR(EINVAL); > > + > > + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel- > > >frame_priv_data_size, 0, > > + frames_c > > tx->device_ctx, > > + hwaccel- > > >free_frame_priv); > > + } else { > > + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel- > > >frame_priv_data_size); > > + } > > + > > if (!*hwaccel_picture_private) > > return AVERROR(ENOMEM); > > > > You never set frames_ctx in this patch. Thank you for the catch. Yep, I did a type doing v3. Fixed now. See v5. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Nov 21, 2023, 07:02 by dmitry.v.rogozhkin-at-intel.com@ffmpeg.org: > On Tue, 2023-11-21 at 06:17 +0100, Lynne wrote: > >> Nov 21, 2023, 05:38 by dmitry.v.rogozhkin-at-intel.com@ffmpeg.org: >> >> > Validate that a hw_frames_ctx is available before using it for >> > the AVHWAccel.free_frame_priv callback, and don't require it to >> > be present when the callback is not in use by the HWAccel. >> > >> > v2: check for free_frame_priv (Hendrik) >> > v3: return EINVAL (Christoph Reiter) >> > v4: better commit message (Hendrik) >> > >> > See[1]: https://github.com/msys2/MINGW-packages/pull/19050 >> > Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv >> > callback") >> > CC: Lynne <dev@lynne.ee> >> > CC: Christoph Reiter <reiter.christoph@gmail.com> >> > Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> >> > --- >> > libavcodec/decode.c | 18 +++++++++++++----- >> > 1 file changed, 13 insertions(+), 5 deletions(-) >> > >> > diff --git a/libavcodec/decode.c b/libavcodec/decode.c >> > index ad39021..50c3995 100644 >> > --- a/libavcodec/decode.c >> > +++ b/libavcodec/decode.c >> > @@ -1838,17 +1838,25 @@ int ff_copy_palette(void *dst, const >> > AVPacket *src, void *logctx) >> > int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void >> > **hwaccel_picture_private) >> > { >> > const FFHWAccel *hwaccel = ffhwaccel(avctx->hwaccel); >> > - AVHWFramesContext *frames_ctx; >> > >> > if (!hwaccel || !hwaccel->frame_priv_data_size) >> > return 0; >> > >> > av_assert0(!*hwaccel_picture_private); >> > >> > - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; >> > - *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel- >> > >frame_priv_data_size, 0, >> > - frames_ctx- >> > >device_ctx, >> > - hwaccel- >> > >free_frame_priv); >> > + if (hwaccel->free_frame_priv) { >> > + AVHWFramesContext *frames_ctx; >> > + >> > + if (!avctx->hw_frames_ctx) >> > + return AVERROR(EINVAL); >> > + >> > + *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel- >> > >frame_priv_data_size, 0, >> > + frames_c >> > tx->device_ctx, >> > + hwaccel- >> > >free_frame_priv); >> > + } else { >> > + *hwaccel_picture_private = ff_refstruct_allocz(hwaccel- >> > >frame_priv_data_size); >> > + } >> > + >> > if (!*hwaccel_picture_private) >> > return AVERROR(ENOMEM); >> > >> >> You never set frames_ctx in this patch. >> > > Thank you for the catch. Yep, I did a type doing v3. Fixed now. See v5. > Pushed, thanks
diff --git a/libavcodec/decode.c b/libavcodec/decode.c index ad39021..3caaeec 100644 --- a/libavcodec/decode.c +++ b/libavcodec/decode.c @@ -1845,9 +1845,9 @@ int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_pr av_assert0(!*hwaccel_picture_private); - frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; + frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx ? avctx->hw_frames_ctx->data : NULL; *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0, - frames_ctx->device_ctx, + frames_ctx ? frames_ctx->device_ctx : NULL, hwaccel->free_frame_priv); if (!*hwaccel_picture_private) return AVERROR(ENOMEM);
Gurd against segfault running VLC decoding under msys2 [1]: Thread 33 received signal SIGSEGV, Segmentation fault. [Switching to Thread 37728.0xadd0] ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00) at libavcodec/decode.c:1848 1848 frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data; (gdb) bt at libavcodec/decode.c:1848 at libavcodec/h264_slice.c:208 first_slice=1) at libavcodec/h264_slice.c:1599 at libavcodec/h264_slice.c:2130 at libavcodec/h264dec.c:652 got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048 (gdb) p avctx $1 = (AVCodecContext *) 0x6447b00 (gdb) p avctx->hw_frames_ctx $2 = (AVBufferRef *) 0x0 See[1]: https://github.com/msys2/MINGW-packages/pull/19050 Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback") CC: Lynne <dev@lynne.ee> CC: Christoph Reiter <reiter.christoph@gmail.com> Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com> --- libavcodec/decode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)