diff mbox series

[FFmpeg-devel] avdevice/lavfi: output wrapped AVFrames

Message ID 20220630152206.15366-1-timo@rothenpieler.org
State New
Headers show
Series [FFmpeg-devel] avdevice/lavfi: output wrapped AVFrames | expand

Checks

Context Check Description
yinshiyou/make_loongarch64 success Make finished
yinshiyou/make_fate_loongarch64 success Make fate finished
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished
andriy/make_armv7_RPi4 success Make finished
andriy/make_fate_armv7_RPi4 success Make fate finished

Commit Message

Timo Rothenpieler June 30, 2022, 3:22 p.m. UTC 
This avoids an extra copy of potentially quite big video frames.
Instead of copying the entire frames data into a rawvideo packet it
packs the frame into a wrapped avframe packet and passes it through
as-is.
Unfortunately, wrapped avframes are set up to be video frames, so the
audio frames unfortunately continue to be copied.

Additionally, this enabled passing through video frames that previously
were impossible to process, like hardware frames or other special
formats that couldn't be packed into a rawvideo packet.
---
 libavdevice/lavfi.c                       | 87 +++++++++--------------
 tests/ref/fate/filter-metadata-cropdetect |  3 +-
 2 files changed, 36 insertions(+), 54 deletions(-)

Comments

Michael Niedermayer July 2, 2022, 8:54 a.m. UTC | #1 
On Thu, Jun 30, 2022 at 05:22:06PM +0200, Timo Rothenpieler wrote:
> This avoids an extra copy of potentially quite big video frames.
> Instead of copying the entire frames data into a rawvideo packet it
> packs the frame into a wrapped avframe packet and passes it through
> as-is.
> Unfortunately, wrapped avframes are set up to be video frames, so the
> audio frames unfortunately continue to be copied.
> 
> Additionally, this enabled passing through video frames that previously
> were impossible to process, like hardware frames or other special
> formats that couldn't be packed into a rawvideo packet.
> ---
>  libavdevice/lavfi.c                       | 87 +++++++++--------------
>  tests/ref/fate/filter-metadata-cropdetect |  3 +-
>  2 files changed, 36 insertions(+), 54 deletions(-)

seems to get this stuck:
./ffmpeg -f lavfi -i 'amovie=fate-suite/wavpack/num_channels/eva_2.22_6.1_16bit-partial.wv,asplit=3[out1][a][b]; [a]showwaves=s=340x240,pad=iw:ih*2[waves]; [b]showspectrum=s=340x240[spectrum]; [waves][spectrum] overlay=0:h [out0]'   file-waves.avi

(stuck as in OOM killed)

thx

[...]
Paul B Mahol July 2, 2022, 12:34 p.m. UTC | #2 
On Sat, Jul 2, 2022 at 10:55 AM Michael Niedermayer <michael@niedermayer.cc>
wrote:

> On Thu, Jun 30, 2022 at 05:22:06PM +0200, Timo Rothenpieler wrote:
> > This avoids an extra copy of potentially quite big video frames.
> > Instead of copying the entire frames data into a rawvideo packet it
> > packs the frame into a wrapped avframe packet and passes it through
> > as-is.
> > Unfortunately, wrapped avframes are set up to be video frames, so the
> > audio frames unfortunately continue to be copied.
> >
> > Additionally, this enabled passing through video frames that previously
> > were impossible to process, like hardware frames or other special
> > formats that couldn't be packed into a rawvideo packet.
> > ---
> >  libavdevice/lavfi.c                       | 87 +++++++++--------------
> >  tests/ref/fate/filter-metadata-cropdetect |  3 +-
> >  2 files changed, 36 insertions(+), 54 deletions(-)
>
> seems to get this stuck:
> ./ffmpeg -f lavfi -i
> 'amovie=fate-suite/wavpack/num_channels/eva_2.22_6.1_16bit-partial.wv,asplit=3[out1][a][b];
> [a]showwaves=s=340x240,pad=iw:ih*2[waves];
> [b]showspectrum=s=340x240[spectrum]; [waves][spectrum] overlay=0:h [out0]'
>  file-waves.avi
>
> (stuck as in OOM killed)
>

OOM killed or its is slow just because of showspectrum filter not
supporting arbitrary sizes in realt time?.

>
> thx
>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Nations do behave wisely once they have exhausted all other alternatives.
> -- Abba Eban
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
Michael Niedermayer July 2, 2022, 12:44 p.m. UTC | #3 
On Sat, Jul 02, 2022 at 02:34:17PM +0200, Paul B Mahol wrote:
> On Sat, Jul 2, 2022 at 10:55 AM Michael Niedermayer <michael@niedermayer.cc>
> wrote:
> 
> > On Thu, Jun 30, 2022 at 05:22:06PM +0200, Timo Rothenpieler wrote:
> > > This avoids an extra copy of potentially quite big video frames.
> > > Instead of copying the entire frames data into a rawvideo packet it
> > > packs the frame into a wrapped avframe packet and passes it through
> > > as-is.
> > > Unfortunately, wrapped avframes are set up to be video frames, so the
> > > audio frames unfortunately continue to be copied.
> > >
> > > Additionally, this enabled passing through video frames that previously
> > > were impossible to process, like hardware frames or other special
> > > formats that couldn't be packed into a rawvideo packet.
> > > ---
> > >  libavdevice/lavfi.c                       | 87 +++++++++--------------
> > >  tests/ref/fate/filter-metadata-cropdetect |  3 +-
> > >  2 files changed, 36 insertions(+), 54 deletions(-)
> >
> > seems to get this stuck:
> > ./ffmpeg -f lavfi -i
> > 'amovie=fate-suite/wavpack/num_channels/eva_2.22_6.1_16bit-partial.wv,asplit=3[out1][a][b];
> > [a]showwaves=s=340x240,pad=iw:ih*2[waves];
> > [b]showspectrum=s=340x240[spectrum]; [waves][spectrum] overlay=0:h [out0]'
> >  file-waves.avi
> >
> > (stuck as in OOM killed)
> >
> 
> OOM killed or its is slow just because of showspectrum filter not
> supporting arbitrary sizes in realt time?.

OOM killed on system with 25gb available
it took a few seconds to get killed, i assume it needed some time to allocate enough.
that system has swap disabled

thx

[...]
Timo Rothenpieler July 5, 2022, 11:26 a.m. UTC | #4 
On 02.07.2022 10:54, Michael Niedermayer wrote:
> seems to get this stuck:
> ./ffmpeg -f lavfi -i 'amovie=fate-suite/wavpack/num_channels/eva_2.22_6.1_16bit-partial.wv,asplit=3[out1][a][b]; [a]showwaves=s=340x240,pad=iw:ih*2[waves]; [b]showspectrum=s=340x240[spectrum]; [waves][spectrum] overlay=0:h [out0]'   file-waves.avi
> 
> (stuck as in OOM killed)

Yeah, same thing is happening here. Very odd for sure.
Seems like it's just buffering an endless amount of frames, without ever 
starting to read and output them.
No idea why (yet)
diff mbox series

Patch

diff --git a/libavdevice/lavfi.c b/libavdevice/lavfi.c
index db5d0b94de..61076f9950 100644
--- a/libavdevice/lavfi.c
+++ b/libavdevice/lavfi.c
@@ -54,32 +54,10 @@  typedef struct {
     int *sink_eof;
     int *stream_sink_map;
     int *sink_stream_subcc_map;
-    AVFrame *decoded_frame;
     int nb_sinks;
     AVPacket subcc_packet;
 } LavfiContext;
 
-static int *create_all_formats(int n)
-{
-    int i, j, *fmts, count = 0;
-
-    for (i = 0; i < n; i++) {
-        const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(i);
-        if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL))
-            count++;
-    }
-
-    if (!(fmts = av_malloc_array(count + 1, sizeof(*fmts))))
-        return NULL;
-    for (j = 0, i = 0; i < n; i++) {
-        const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(i);
-        if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL))
-            fmts[j++] = i;
-    }
-    fmts[j] = AV_PIX_FMT_NONE;
-    return fmts;
-}
-
 av_cold static int lavfi_read_close(AVFormatContext *avctx)
 {
     LavfiContext *lavfi = avctx->priv_data;
@@ -90,7 +68,6 @@  av_cold static int lavfi_read_close(AVFormatContext *avctx)
     av_freep(&lavfi->sink_stream_subcc_map);
     av_freep(&lavfi->sinks);
     avfilter_graph_free(&lavfi->graph);
-    av_frame_free(&lavfi->decoded_frame);
 
     return 0;
 }
@@ -125,15 +102,11 @@  av_cold static int lavfi_read_header(AVFormatContext *avctx)
     LavfiContext *lavfi = avctx->priv_data;
     AVFilterInOut *input_links = NULL, *output_links = NULL, *inout;
     const AVFilter *buffersink, *abuffersink;
-    int *pix_fmts = create_all_formats(AV_PIX_FMT_NB);
     enum AVMediaType type;
     int ret = 0, i, n;
 
 #define FAIL(ERR) { ret = ERR; goto end; }
 
-    if (!pix_fmts)
-        FAIL(AVERROR(ENOMEM));
-
     buffersink = avfilter_get_by_name("buffersink");
     abuffersink = avfilter_get_by_name("abuffersink");
 
@@ -264,8 +237,6 @@  av_cold static int lavfi_read_header(AVFormatContext *avctx)
             ret = avfilter_graph_create_filter(&sink, buffersink,
                                                inout->name, NULL,
                                                NULL, lavfi->graph);
-            if (ret >= 0)
-                ret = av_opt_set_int_list(sink, "pix_fmts", pix_fmts,  AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
             if (ret < 0)
                 goto end;
         } else if (type == AVMEDIA_TYPE_AUDIO) {
@@ -321,15 +292,11 @@  av_cold static int lavfi_read_header(AVFormatContext *avctx)
         avpriv_set_pts_info(st, 64, time_base.num, time_base.den);
         par->codec_type = av_buffersink_get_type(sink);
         if (par->codec_type == AVMEDIA_TYPE_VIDEO) {
-            int64_t probesize;
-            par->codec_id   = AV_CODEC_ID_RAWVIDEO;
+            par->codec_id   = AV_CODEC_ID_WRAPPED_AVFRAME;
             par->format     = av_buffersink_get_format(sink);
             par->width      = av_buffersink_get_w(sink);
             par->height     = av_buffersink_get_h(sink);
-            probesize       = par->width * par->height * 30 *
-                              av_get_padded_bits_per_pixel(av_pix_fmt_desc_get(par->format));
-            avctx->probesize = FFMAX(avctx->probesize, probesize);
-            st       ->sample_aspect_ratio =
+            st ->sample_aspect_ratio =
             par->sample_aspect_ratio = av_buffersink_get_sample_aspect_ratio(sink);
         } else if (par->codec_type == AVMEDIA_TYPE_AUDIO) {
             par->sample_rate = av_buffersink_get_sample_rate(sink);
@@ -348,11 +315,7 @@  av_cold static int lavfi_read_header(AVFormatContext *avctx)
     if ((ret = create_subcc_streams(avctx)) < 0)
         goto end;
 
-    if (!(lavfi->decoded_frame = av_frame_alloc()))
-        FAIL(AVERROR(ENOMEM));
-
 end:
-    av_free(pix_fmts);
     avfilter_inout_free(&input_links);
     avfilter_inout_free(&output_links);
     return ret;
@@ -378,15 +341,20 @@  static int create_subcc_packet(AVFormatContext *avctx, AVFrame *frame,
     return 0;
 }
 
+static void lavfi_free_frame(void *opaque, uint8_t *data)
+{
+    AVFrame *frame = (AVFrame*)data;
+    av_frame_free(&frame);
+}
+
 static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
 {
     LavfiContext *lavfi = avctx->priv_data;
     double min_pts = DBL_MAX;
     int stream_idx, min_pts_sink_idx = 0;
-    AVFrame *frame = lavfi->decoded_frame;
+    AVFrame *frame;
     AVDictionary *frame_metadata;
     int ret, i;
-    int size = 0;
     AVStream *st;
 
     if (lavfi->subcc_packet.size) {
@@ -394,6 +362,10 @@  static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
         return pkt->size;
     }
 
+    frame = av_frame_alloc();
+    if (!frame)
+        return AVERROR(ENOMEM);
+
     /* iterate through all the graph sinks. Select the sink with the
      * minimum PTS */
     for (i = 0; i < lavfi->nb_sinks; i++) {
@@ -411,7 +383,7 @@  static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
             lavfi->sink_eof[i] = 1;
             continue;
         } else if (ret < 0)
-            return ret;
+            goto fail;
         d = av_rescale_q_rnd(frame->pts, tb, AV_TIME_BASE_Q, AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX);
         ff_dlog(avctx, "sink_idx:%d time:%f\n", i, d);
         av_frame_unref(frame);
@@ -421,8 +393,10 @@  static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
             min_pts_sink_idx = i;
         }
     }
-    if (min_pts == DBL_MAX)
-        return AVERROR_EOF;
+    if (min_pts == DBL_MAX) {
+        ret = AVERROR_EOF;
+        goto fail;
+    }
 
     ff_dlog(avctx, "min_pts_sink_idx:%i\n", min_pts_sink_idx);
 
@@ -431,15 +405,19 @@  static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
     st = avctx->streams[stream_idx];
 
     if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
-        size = av_image_get_buffer_size(frame->format, frame->width, frame->height, 1);
-        if ((ret = av_new_packet(pkt, size)) < 0)
+        pkt->buf = av_buffer_create((uint8_t*)frame, sizeof(*frame),
+                                    &lavfi_free_frame, NULL, 0);
+        if (!pkt->buf) {
+            ret = AVERROR(ENOMEM);
             goto fail;
+        }
 
-        av_image_copy_to_buffer(pkt->data, size, (const uint8_t **)frame->data, frame->linesize,
-                                frame->format, frame->width, frame->height, 1);
+        pkt->data   = pkt->buf->data;
+        pkt->size   = pkt->buf->size;
+        pkt->flags |= AV_PKT_FLAG_TRUSTED;
     } else if (st->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
-        size = frame->nb_samples * av_get_bytes_per_sample(frame->format) *
-                                   frame->ch_layout.nb_channels;
+        int size = frame->nb_samples * av_get_bytes_per_sample(frame->format) *
+                                       frame->ch_layout.nb_channels;
         if ((ret = av_new_packet(pkt, size)) < 0)
             goto fail;
         memcpy(pkt->data, frame->data[0], size);
@@ -468,10 +446,13 @@  static int lavfi_read_packet(AVFormatContext *avctx, AVPacket *pkt)
     pkt->stream_index = stream_idx;
     pkt->pts = frame->pts;
     pkt->pos = frame->pkt_pos;
-    av_frame_unref(frame);
-    return size;
+
+    if (st->codecpar->codec_type != AVMEDIA_TYPE_VIDEO)
+        av_frame_free(&frame);
+
+    return pkt->size;
 fail:
-    av_frame_unref(frame);
+    av_frame_free(&frame);
     return ret;
 
 }
diff --git a/tests/ref/fate/filter-metadata-cropdetect b/tests/ref/fate/filter-metadata-cropdetect
index e7c03ead4e..104acdd77f 100644
--- a/tests/ref/fate/filter-metadata-cropdetect
+++ b/tests/ref/fate/filter-metadata-cropdetect
@@ -1,4 +1,5 @@ 
-pts=0
+pts=0|
+
 pts=400
 pts=800|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=719|tag:lavfi.cropdetect.y1=61|tag:lavfi.cropdetect.y2=424|tag:lavfi.cropdetect.w=720|tag:lavfi.cropdetect.h=352|tag:lavfi.cropdetect.x=0|tag:lavfi.cropdetect.y=68
 pts=1200|tag:lavfi.cropdetect.x1=0|tag:lavfi.cropdetect.x2=719|tag:lavfi.cropdetect.y1=61|tag:lavfi.cropdetect.y2=424|tag:lavfi.cropdetect.w=720|tag:lavfi.cropdetect.h=352|tag:lavfi.cropdetect.x=0|tag:lavfi.cropdetect.y=68