diff mbox series

[FFmpeg-devel] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not available. now filter fails if secondary frame is not available due to e.g secondary source delay.

Message ID CAAE7vKdESTfrXpfS_bPGmskYaXxnUhBpV6cHBQsH8ZNq_m9opw@mail.gmail.com
State New
Headers show
Series [FFmpeg-devel] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not available. now filter fails if secondary frame is not available due to e.g secondary source delay.
Related show

Checks

Context Check Description
andriy/default pending
andriy/make success Make finished
andriy/make_fate success Make fate finished

Commit Message

Alex Pokotilo Aug. 21, 2020, 9:41 a.m. UTC
Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not
 available. now filter fails if secondary frame is not available due to e.g secondary source delay.

---
 libavfilter/vf_overlay_cuda.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Nicolas George Aug. 21, 2020, 9:54 a.m. UTC | #1
Alex Pokotilo (12020-08-21):
> From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001
> From: Alex Pokotilo <support@wmspanel.com>
> Date: Fri, 21 Aug 2020 09:14:37 +0000
> Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not
>  available. now filter fails if secondary frame is not available due to e.g secondary source delay.

Your Git commit message is badly formatted. The format is:

	context: short summary on one line

	After an empty line, details if necessary, on several lines and
	wrapped at a reasonable line length.

The fix may be correct, but your analysis of the problem does not prove
it. Unless there is a serious bug in framesync, it will not call
overlay_cuda_blend() without a secondary frame "due to e.g secondary
source delay.", it will only do so if the stream has not yet begun or
has ended.

Please give more details on the exact circumstances you observe the
issue.

Regards,
Alex Pokotilo Aug. 21, 2020, 10:46 a.m. UTC | #2
On Fri, Aug 21, 2020 at 7:54 PM Nicolas George <george@nsup.org> wrote:
>
> Alex Pokotilo (12020-08-21):
> > From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001
> > From: Alex Pokotilo <support@wmspanel.com>
> > Date: Fri, 21 Aug 2020 09:14:37 +0000
> > Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not
> >  available. now filter fails if secondary frame is not available due to e.g secondary source delay.
>
> Your Git commit message is badly formatted. The format is:
>
>         context: short summary on one line
>
>         After an empty line, details if necessary, on several lines and
>         wrapped at a reasonable line length.
>

I'm sorry. I'm a newbie. Am I right I just need to commit in following format:

first line: short description......
LF
more details here

Should I send a new patch as a separate email or reply to this one ?
Sorry again.
> The fix may be correct, but your analysis of the problem does not prove
> it. Unless there is a serious bug in framesync, it will not call
> overlay_cuda_blend() without a secondary frame "due to e.g secondary
> source delay.", it will only do so if the stream has not yet begun or
> has ended.
>
> Please give more details on the exact circumstances you observe the
> issue.

I create a filter graph with vf_cuda_overlay in my application. I
start sending frames into a "main" stream link using
av_buffersrc_write_frame.
I start sending frames into a "secondary" stream too almost
immediately. I have one output sink in my filter graph.
After each write I call av_buffersink_get_frame and get AVERROR_BUG
from vf_overlay_cuda for some time initially but after several frames
it starts returning mixed frame.
I've checked vf_blend and other framesync filters. They do the same as
I did in the fix.

Maybe the problem In my application I cannot guarantee that both
streams have the same timeline.
Could you please point me to the place where framesync checks second
stream availability before it calls it's filter(overlay_cuda_blend in
my case).
This topic is very complex to me now so any help from your side can
really help me.
Alexander Strasser Aug. 21, 2020, 10:55 a.m. UTC | #3
On 2020-08-21 20:46 +1000, Alex Pokotilo wrote:
> On Fri, Aug 21, 2020 at 7:54 PM Nicolas George <george@nsup.org> wrote:
> >
> > Alex Pokotilo (12020-08-21):
> > > From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001
> > > From: Alex Pokotilo <support@wmspanel.com>
> > > Date: Fri, 21 Aug 2020 09:14:37 +0000
> > > Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not
> > >  available. now filter fails if secondary frame is not available due to e.g secondary source delay.
> >
> > Your Git commit message is badly formatted. The format is:
> >
> >         context: short summary on one line
> >
> >         After an empty line, details if necessary, on several lines and
> >         wrapped at a reasonable line length.
> >
>
> I'm sorry. I'm a newbie. Am I right I just need to commit in following format:
>
> first line: short description......
> LF
> more details here

Yes, that should do the trick.

Please don't forget to mention the "Why" part in the commit message
body (the "more details here" part), if you re-do the patch.

At the moment it would probably best to first wait for feed back on
your explanation below.


  Alexander

> Should I send a new patch as a separate email or reply to this one ?
> Sorry again.
> > The fix may be correct, but your analysis of the problem does not prove
> > it. Unless there is a serious bug in framesync, it will not call
> > overlay_cuda_blend() without a secondary frame "due to e.g secondary
> > source delay.", it will only do so if the stream has not yet begun or
> > has ended.
> >
> > Please give more details on the exact circumstances you observe the
> > issue.
>
> I create a filter graph with vf_cuda_overlay in my application. I
> start sending frames into a "main" stream link using
> av_buffersrc_write_frame.
> I start sending frames into a "secondary" stream too almost
> immediately. I have one output sink in my filter graph.
> After each write I call av_buffersink_get_frame and get AVERROR_BUG
> from vf_overlay_cuda for some time initially but after several frames
> it starts returning mixed frame.
> I've checked vf_blend and other framesync filters. They do the same as
> I did in the fix.
>
> Maybe the problem In my application I cannot guarantee that both
> streams have the same timeline.
> Could you please point me to the place where framesync checks second
> stream availability before it calls it's filter(overlay_cuda_blend in
> my case).
> This topic is very complex to me now so any help from your side can
> really help me.
Nicolas George Aug. 21, 2020, 11:01 a.m. UTC | #4
Alex Pokotilo (12020-08-21):
> I'm sorry. I'm a newbie. Am I right I just need to commit in following format:

No need to apologize.

> first line: short description......
> LF
> more details here

Exactly.

> Should I send a new patch as a separate email or reply to this one ?

It does not matter.

> I create a filter graph with vf_cuda_overlay in my application. I
> start sending frames into a "main" stream link using
> av_buffersrc_write_frame.
> I start sending frames into a "secondary" stream too almost
> immediately. I have one output sink in my filter graph.

What are the timestamps (and corresponding time bases) of the first
frame you send on each input?

> After each write I call av_buffersink_get_frame and get AVERROR_BUG
> from vf_overlay_cuda for some time initially but after several frames
> it starts returning mixed frame.
> I've checked vf_blend and other framesync filters. They do the same as
> I did in the fix.

The fix is probably right, but I would like to make 100% sure, and I do
not have a CUDA setup. Plus, it would help check the code of your
application is right.

> Maybe the problem In my application I cannot guarantee that both
> streams have the same timeline.
> Could you please point me to the place where framesync checks second
> stream availability before it calls it's filter(overlay_cuda_blend in
> my case).
> This topic is very complex to me now so any help from your side can
> really help me.

The code in framesync is rather complex because it is designed to handle
a wide variety of cases.

If you are willing to make a quick test, this would be interesting: in
libavfilter/framesync.c, in the function ff_framesync_init_dualinput,
replace fs->in[1].before = EXT_NULL; with EXT_INFINITY and see if it
changes something.

But most importantly, seeing the timestamps of the frames would let me
know if the issue is exactly what I think it is.

Regards,
Alex Pokotilo Aug. 24, 2020, 4:25 a.m. UTC | #5
>
> What are the timestamps (and corresponding time bases) of the first
> frame you send on each input?

this frame is sent the first
fs->in[0]
 pts = 0;
 timscale=1/1000 but it doesn't matter as pts is 0

this frame is sent the second
fs->in[1]
 pts =140
 timscale=1/25

When I call av_buffersink_get_frame after sending the frame into fs->in[1]
 i see only the main_frame in overlay_cuda_blend function.
So in my case I send the main frame with pts < pts of the second frame.

> If you are willing to make a quick test, this would be interesting: in
> libavfilter/framesync.c, in the function ff_framesync_init_dualinput,
> replace fs->in[1].before = EXT_NULL; with EXT_INFINITY and see if it
> changes something.
When I change code as proposed I always see a main frame with input_overlay.
I was able to find why it works after the fix checking
framesync_advance function code.


One observation not related to vf_overlay_cude fix if you don't mind:
even with the fix you proposed if I don't send a single frame into
fs->in[1](i.e secondary stream) I cannot get any output.
This is because consume_from_fifos doesn't honor .before ==
EXT_INFINITY and .state == STATE_BOF

and requires at least one frame for each input.
I cannot say is it WAD or not - just my observation.
When I read a description about EXT_INFINITY I guessed this flag
should force processing other inputs ignoring this one. But again if I
am wrong(and most probably I missed something) just ignore this.
Alex Pokotilo Aug. 24, 2020, 2:50 p.m. UTC | #6
> One observation not related to vf_overlay_cude fix if you don't mind:
> even with the fix you proposed if I don't send a single frame into
> fs->in[1](i.e secondary stream) I cannot get any output.
> This is because consume_from_fifos doesn't honor .before ==
> EXT_INFINITY and .state == STATE_BOF
>

let me correct myself: consume_from_fifos doesn't honor .before ==
EXT_NULL and .state == STATE_BOF

> and requires at least one frame for each input.
> I cannot say is it WAD or not - just my observation.
> When I read a description about EXT_INFINITY I guessed this flag
> should force processing other inputs ignoring this one

Again let me correct myself here too:  I think that if I set
fs->in[1].before== EXT_NULL (this is default behavior)
framesync should return main frames even if I don't send any frames
into fs->in[1].
but consume_from_fifos doesn't honor .before == EXT_NULL and .state ==
STATE_BOF and this is why it waits for fs->in[1] frames forever.
Sorry for bothering you with unrelated question
diff mbox series

Patch

diff --git a/libavfilter/vf_overlay_cuda.c b/libavfilter/vf_overlay_cuda.c
index 2f0f860e50..355cb39019 100644
--- a/libavfilter/vf_overlay_cuda.c
+++ b/libavfilter/vf_overlay_cuda.c
@@ -157,8 +157,8 @@  static int overlay_cuda_blend(FFFrameSync *fs)
     if (ret < 0)
         return ret;
 
-    if (!input_main || !input_overlay)
-        return AVERROR_BUG;
+    if (!input_overlay)
+        return ff_filter_frame(outlink, input_main);
 
     ret = av_frame_make_writable(input_main);
     if (ret < 0) {