diff mbox series

[FFmpeg-devel,1/6] Revert "avcodec: add FF_CODEC_CAP_INIT_CLEANUP for all codecs which use ff_mpv_common_init()"

Message ID HE1PR0301MB21541C25371C36A1A46E2AB58F779@HE1PR0301MB2154.eurprd03.prod.outlook.com
State Accepted
Headers show
Series [FFmpeg-devel,1/6] Revert "avcodec: add FF_CODEC_CAP_INIT_CLEANUP for all codecs which use ff_mpv_common_init()" | expand

Checks

Context Check Description
andriy/x86_make success Make finished
andriy/x86_make_fate success Make fate finished
andriy/PPC64_make success Make finished
andriy/PPC64_make_fate success Make fate finished

Commit Message

Andreas Rheinhardt April 5, 2021, 1:28 a.m. UTC 
From: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>

This mostly reverts commit 4b2863ff01b1fe93d9a518523c9098d17a9d8c6f.
Said commit removed the freeing code from ff_mpv_common_init(),
ff_mpv_common_frame_size_change() and ff_mpeg_framesize_alloc() and
instead added the FF_CODEC_CAP_INIT_CLEANUP to several codecs that use
ff_mpv_common_init(). This introduced several bugs:

a) Several decoders using ff_mpv_common_init() in their init function were
forgotten: This affected FLV, Intel H.263, RealVideo 3.0 and V4.0 as well as
VC-1/WMV3.
b) ff_mpv_common_init() is not only called from the init function of
codecs, it is also called from AVCodec.decode functions. If an error
happens after an allocation has succeeded, it can lead to memleaks;
furthermore, it is now possible for the MpegEncContext to be marked as
initialized even when ff_mpv_common_init() returns an error and this can
lead to segfaults because decoders that call ff_mpv_common_init() when
decoding a frame can mistakenly think that the MpegEncContext has been
properly initialized. This can e.g. happen with H.261 or MPEG-4.
c) Removing code for freeing from ff_mpeg_framesize_alloc() (which can't
be called from any init function) can lead to segfaults because the
check for whether it needs to allocate consists of checking whether the
first of the buffers allocated there has been allocated. This part has
already been fixed in 76cea1d2ce3f23e8131c8664086a1daf873ed694.
d) ff_mpv_common_frame_size_change() can also not be reached from any
AVCodec.init function; yet the changes can e.g. lead to segfaults with
decoders using ff_h263_decode_frame() upon allocation failure, because
the MpegEncContext will upon return be flagged as both initialized and
not in need of reinitialization (granted, the fact that
ff_h263_decode_frame() clears context_reinit before the context has been
reinited is a bug in itself). With the earlier version, the context
would be cleaned upon failure and it would be attempted to initialize
the context again in the next call to ff_h263_decode_frame().

While a) could be fixed by adding the missing FF_CODEC_CAP_INIT_CLEANUP,
keeping the current approach would entail adding cleanup code to several
other places because of b). Therefore ff_mpv_common_init() is again made
to clean up after itself; the changes to the wmv2 decoder and the SVQ1
encoder have not been reverted: The former fixed a memleak, the latter
allowed to remove cleanup code.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
---
 libavcodec/h261dec.c       |  2 +-
 libavcodec/h263dec.c       |  4 ++--
 libavcodec/mpeg12dec.c     |  9 +++++----
 libavcodec/mpeg4videodec.c |  3 +--
 libavcodec/mpegvideo.c     | 31 ++++++++++++++++++++-----------
 libavcodec/msmpeg4dec.c    |  8 ++++----
 libavcodec/rv10.c          |  2 --
 7 files changed, 33 insertions(+), 26 deletions(-)

Comments

Michael Niedermayer April 7, 2021, 7:53 p.m. UTC | #1 
On Mon, Apr 05, 2021 at 03:28:52AM +0200, Andreas Rheinhardt wrote:
> From: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
> 
> This mostly reverts commit 4b2863ff01b1fe93d9a518523c9098d17a9d8c6f.
> Said commit removed the freeing code from ff_mpv_common_init(),
> ff_mpv_common_frame_size_change() and ff_mpeg_framesize_alloc() and
> instead added the FF_CODEC_CAP_INIT_CLEANUP to several codecs that use
> ff_mpv_common_init(). This introduced several bugs:
> 
> a) Several decoders using ff_mpv_common_init() in their init function were
> forgotten: This affected FLV, Intel H.263, RealVideo 3.0 and V4.0 as well as
> VC-1/WMV3.
> b) ff_mpv_common_init() is not only called from the init function of
> codecs, it is also called from AVCodec.decode functions. If an error
> happens after an allocation has succeeded, it can lead to memleaks;
> furthermore, it is now possible for the MpegEncContext to be marked as
> initialized even when ff_mpv_common_init() returns an error and this can
> lead to segfaults because decoders that call ff_mpv_common_init() when
> decoding a frame can mistakenly think that the MpegEncContext has been
> properly initialized. This can e.g. happen with H.261 or MPEG-4.
> c) Removing code for freeing from ff_mpeg_framesize_alloc() (which can't
> be called from any init function) can lead to segfaults because the
> check for whether it needs to allocate consists of checking whether the
> first of the buffers allocated there has been allocated. This part has
> already been fixed in 76cea1d2ce3f23e8131c8664086a1daf873ed694.
> d) ff_mpv_common_frame_size_change() can also not be reached from any
> AVCodec.init function; yet the changes can e.g. lead to segfaults with
> decoders using ff_h263_decode_frame() upon allocation failure, because
> the MpegEncContext will upon return be flagged as both initialized and
> not in need of reinitialization (granted, the fact that
> ff_h263_decode_frame() clears context_reinit before the context has been
> reinited is a bug in itself). With the earlier version, the context
> would be cleaned upon failure and it would be attempted to initialize
> the context again in the next call to ff_h263_decode_frame().
> 
> While a) could be fixed by adding the missing FF_CODEC_CAP_INIT_CLEANUP,
> keeping the current approach would entail adding cleanup code to several
> other places because of b). Therefore ff_mpv_common_init() is again made
> to clean up after itself; the changes to the wmv2 decoder and the SVQ1
> encoder have not been reverted: The former fixed a memleak, the latter
> allowed to remove cleanup code.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
> ---
>  libavcodec/h261dec.c       |  2 +-
>  libavcodec/h263dec.c       |  4 ++--
>  libavcodec/mpeg12dec.c     |  9 +++++----
>  libavcodec/mpeg4videodec.c |  3 +--
>  libavcodec/mpegvideo.c     | 31 ++++++++++++++++++++-----------
>  libavcodec/msmpeg4dec.c    |  8 ++++----
>  libavcodec/rv10.c          |  2 --
>  7 files changed, 33 insertions(+), 26 deletions(-)

whole patchset LGTM
also seems to work fine

please add mentionings of any CISPA samples this fixes so that is kept track
of. (assuming you succeded reproducing the issues or know/can guess reasonably
well which patch fixed what)

thx

[...]
diff mbox series

Patch

diff --git a/libavcodec/h261dec.c b/libavcodec/h261dec.c
index 5c25aa9cb3..eb544e6046 100644
--- a/libavcodec/h261dec.c
+++ b/libavcodec/h261dec.c
@@ -678,6 +678,6 @@  AVCodec ff_h261_decoder = {
     .close          = h261_decode_end,
     .decode         = h261_decode_frame,
     .capabilities   = AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE,
     .max_lowres     = 3,
 };
diff --git a/libavcodec/h263dec.c b/libavcodec/h263dec.c
index dafa54d8d4..e8b4d83e6e 100644
--- a/libavcodec/h263dec.c
+++ b/libavcodec/h263dec.c
@@ -771,7 +771,7 @@  AVCodec ff_h263_decoder = {
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1 |
                       AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .flush          = ff_mpeg_flush,
     .max_lowres     = 3,
     .pix_fmts       = ff_h263_hwaccel_pixfmt_list_420,
@@ -789,7 +789,7 @@  AVCodec ff_h263p_decoder = {
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1 |
                       AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .flush          = ff_mpeg_flush,
     .max_lowres     = 3,
     .pix_fmts       = ff_h263_hwaccel_pixfmt_list_420,
diff --git a/libavcodec/mpeg12dec.c b/libavcodec/mpeg12dec.c
index 92dd6a0b24..94221da2c1 100644
--- a/libavcodec/mpeg12dec.c
+++ b/libavcodec/mpeg12dec.c
@@ -2880,7 +2880,8 @@  static av_cold int mpeg_decode_end(AVCodecContext *avctx)
 {
     Mpeg1Context *s = avctx->priv_data;
 
-    ff_mpv_common_end(&s->mpeg_enc_ctx);
+    if (s->mpeg_enc_ctx_allocated)
+        ff_mpv_common_end(&s->mpeg_enc_ctx);
     av_buffer_unref(&s->a53_buf_ref);
     return 0;
 }
@@ -2897,7 +2898,7 @@  AVCodec ff_mpeg1video_decoder = {
     .capabilities          = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1 |
                              AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY |
                              AV_CODEC_CAP_SLICE_THREADS,
-    .caps_internal         = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP |
+    .caps_internal         = FF_CODEC_CAP_INIT_THREADSAFE |
                              FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .flush                 = flush,
     .max_lowres            = 3,
@@ -2931,7 +2932,7 @@  AVCodec ff_mpeg2video_decoder = {
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1 |
                       AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY |
                       AV_CODEC_CAP_SLICE_THREADS,
-    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP |
+    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE |
                       FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .flush          = flush,
     .max_lowres     = 3,
@@ -2976,7 +2977,7 @@  AVCodec ff_mpegvideo_decoder = {
     .close          = mpeg_decode_end,
     .decode         = mpeg_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1 | AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY | AV_CODEC_CAP_SLICE_THREADS,
-    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE | FF_CODEC_CAP_INIT_CLEANUP |
+    .caps_internal  = FF_CODEC_CAP_INIT_THREADSAFE |
                       FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .flush          = flush,
     .max_lowres     = 3,
diff --git a/libavcodec/mpeg4videodec.c b/libavcodec/mpeg4videodec.c
index 2ca804fdc6..2c440a5026 100644
--- a/libavcodec/mpeg4videodec.c
+++ b/libavcodec/mpeg4videodec.c
@@ -3579,8 +3579,7 @@  AVCodec ff_mpeg4_decoder = {
                              AV_CODEC_CAP_TRUNCATED | AV_CODEC_CAP_DELAY |
                              AV_CODEC_CAP_FRAME_THREADS,
     .caps_internal         = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM |
-                             FF_CODEC_CAP_ALLOCATE_PROGRESS |
-                             FF_CODEC_CAP_INIT_CLEANUP,
+                             FF_CODEC_CAP_ALLOCATE_PROGRESS,
     .flush                 = ff_mpeg_flush,
     .max_lowres            = 3,
     .pix_fmts              = ff_h263_hwaccel_pixfmt_list_420,
diff --git a/libavcodec/mpegvideo.c b/libavcodec/mpegvideo.c
index 728e591958..2351a09874 100644
--- a/libavcodec/mpegvideo.c
+++ b/libavcodec/mpegvideo.c
@@ -932,17 +932,17 @@  av_cold int ff_mpv_common_init(MpegEncContext *s)
     for (i = 0; i < MAX_PICTURE_COUNT; i++) {
         s->picture[i].f = av_frame_alloc();
         if (!s->picture[i].f)
-            return AVERROR(ENOMEM);
+            goto fail_nomem;
     }
 
     if (!(s->next_picture.f    = av_frame_alloc()) ||
         !(s->last_picture.f    = av_frame_alloc()) ||
         !(s->current_picture.f = av_frame_alloc()) ||
         !(s->new_picture.f     = av_frame_alloc()))
-        return AVERROR(ENOMEM);
+        goto fail_nomem;
 
     if ((ret = init_context_frame(s)))
-        return AVERROR(ENOMEM);
+        goto fail;
 
     s->parse_context.state = -1;
 
@@ -956,10 +956,10 @@  av_cold int ff_mpv_common_init(MpegEncContext *s)
             if (i) {
                 s->thread_context[i] = av_memdup(s, sizeof(MpegEncContext));
                 if (!s->thread_context[i])
-                    return AVERROR(ENOMEM);
+                    goto fail_nomem;
             }
             if ((ret = init_duplicate_context(s->thread_context[i])) < 0)
-                return ret;
+                goto fail;
             s->thread_context[i]->start_mb_y =
                 (s->mb_height * (i) + nb_slices / 2) / nb_slices;
             s->thread_context[i]->end_mb_y   =
@@ -967,7 +967,7 @@  av_cold int ff_mpv_common_init(MpegEncContext *s)
         }
     } else {
         if ((ret = init_duplicate_context(s)) < 0)
-            return ret;
+            goto fail;
         s->start_mb_y = 0;
         s->end_mb_y   = s->mb_height;
     }
@@ -975,6 +975,11 @@  av_cold int ff_mpv_common_init(MpegEncContext *s)
 //     }
 
     return 0;
+ fail_nomem:
+    ret = AVERROR(ENOMEM);
+ fail:
+    ff_mpv_common_end(s);
+    return ret;
 }
 
 /**
@@ -1067,7 +1072,7 @@  int ff_mpv_common_frame_size_change(MpegEncContext *s)
 
     if ((s->width || s->height) &&
         (err = av_image_check_size(s->width, s->height, 0, s->avctx)) < 0)
-        return err;
+        goto fail;
 
     /* set chroma shifts */
     err = av_pix_fmt_get_chroma_sub_sample(s->avctx->pix_fmt,
@@ -1077,7 +1082,7 @@  int ff_mpv_common_frame_size_change(MpegEncContext *s)
         return err;
 
     if ((err = init_context_frame(s)))
-        return err;
+        goto fail;
 
     memset(s->thread_context, 0, sizeof(s->thread_context));
     s->thread_context[0]   = s;
@@ -1089,11 +1094,12 @@  int ff_mpv_common_frame_size_change(MpegEncContext *s)
                 if (i) {
                     s->thread_context[i] = av_memdup(s, sizeof(MpegEncContext));
                     if (!s->thread_context[i]) {
-                        return AVERROR(ENOMEM);
+                        err = AVERROR(ENOMEM);
+                        goto fail;
                     }
                 }
                 if ((err = init_duplicate_context(s->thread_context[i])) < 0)
-                    return err;
+                    goto fail;
                 s->thread_context[i]->start_mb_y =
                     (s->mb_height * (i) + nb_slices / 2) / nb_slices;
                 s->thread_context[i]->end_mb_y   =
@@ -1102,7 +1108,7 @@  int ff_mpv_common_frame_size_change(MpegEncContext *s)
         } else {
             err = init_duplicate_context(s);
             if (err < 0)
-                return err;
+                goto fail;
             s->start_mb_y = 0;
             s->end_mb_y   = s->mb_height;
         }
@@ -1110,6 +1116,9 @@  int ff_mpv_common_frame_size_change(MpegEncContext *s)
     }
 
     return 0;
+ fail:
+    ff_mpv_common_end(s);
+    return err;
 }
 
 /* init common structure for both encoder and decoder */
diff --git a/libavcodec/msmpeg4dec.c b/libavcodec/msmpeg4dec.c
index 405fda4b83..63f9faa1bb 100644
--- a/libavcodec/msmpeg4dec.c
+++ b/libavcodec/msmpeg4dec.c
@@ -870,7 +870,7 @@  AVCodec ff_msmpeg4v1_decoder = {
     .close          = ff_h263_decode_end,
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {
         AV_PIX_FMT_YUV420P,
@@ -888,7 +888,7 @@  AVCodec ff_msmpeg4v2_decoder = {
     .close          = ff_h263_decode_end,
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {
         AV_PIX_FMT_YUV420P,
@@ -906,7 +906,7 @@  AVCodec ff_msmpeg4v3_decoder = {
     .close          = ff_h263_decode_end,
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {
         AV_PIX_FMT_YUV420P,
@@ -924,7 +924,7 @@  AVCodec ff_wmv1_decoder = {
     .close          = ff_h263_decode_end,
     .decode         = ff_h263_decode_frame,
     .capabilities   = AV_CODEC_CAP_DRAW_HORIZ_BAND | AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM | FF_CODEC_CAP_INIT_CLEANUP,
+    .caps_internal  = FF_CODEC_CAP_SKIP_FRAME_FILL_PARAM,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {
         AV_PIX_FMT_YUV420P,
diff --git a/libavcodec/rv10.c b/libavcodec/rv10.c
index d5c7480611..89b838ad15 100644
--- a/libavcodec/rv10.c
+++ b/libavcodec/rv10.c
@@ -687,7 +687,6 @@  AVCodec ff_rv10_decoder = {
     .close          = rv10_decode_end,
     .decode         = rv10_decode_frame,
     .capabilities   = AV_CODEC_CAP_DR1,
-    .caps_internal  = FF_CODEC_CAP_INIT_CLEANUP,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {
         AV_PIX_FMT_YUV420P,
@@ -705,7 +704,6 @@  AVCodec ff_rv20_decoder = {
     .close          = rv10_decode_end,
     .decode         = rv10_decode_frame,
     .capabilities   = AV_CODEC_CAP_DR1 | AV_CODEC_CAP_DELAY,
-    .caps_internal  = FF_CODEC_CAP_INIT_CLEANUP,
     .flush          = ff_mpeg_flush,
     .max_lowres     = 3,
     .pix_fmts       = (const enum AVPixelFormat[]) {