| Message ID | CAAE7vKdESTfrXpfS_bPGmskYaXxnUhBpV6cHBQsH8ZNq_m9opw@mail.gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [FFmpeg-devel] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not available. now filter fails if secondary frame is not available due to e.g secondary source delay. | expand |
| Context | Check | Description |
|---|---|---|
| andriy/default | pending | |
| andriy/make | success | Make finished |
| andriy/make_fate | success | Make fate finished |
Alex Pokotilo (12020-08-21): > From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001 > From: Alex Pokotilo <support@wmspanel.com> > Date: Fri, 21 Aug 2020 09:14:37 +0000 > Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not > available. now filter fails if secondary frame is not available due to e.g secondary source delay. Your Git commit message is badly formatted. The format is: context: short summary on one line After an empty line, details if necessary, on several lines and wrapped at a reasonable line length. The fix may be correct, but your analysis of the problem does not prove it. Unless there is a serious bug in framesync, it will not call overlay_cuda_blend() without a secondary frame "due to e.g secondary source delay.", it will only do so if the stream has not yet begun or has ended. Please give more details on the exact circumstances you observe the issue. Regards,
On Fri, Aug 21, 2020 at 7:54 PM Nicolas George <george@nsup.org> wrote: > > Alex Pokotilo (12020-08-21): > > From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001 > > From: Alex Pokotilo <support@wmspanel.com> > > Date: Fri, 21 Aug 2020 09:14:37 +0000 > > Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not > > available. now filter fails if secondary frame is not available due to e.g secondary source delay. > > Your Git commit message is badly formatted. The format is: > > context: short summary on one line > > After an empty line, details if necessary, on several lines and > wrapped at a reasonable line length. > I'm sorry. I'm a newbie. Am I right I just need to commit in following format: first line: short description...... LF more details here Should I send a new patch as a separate email or reply to this one ? Sorry again. > The fix may be correct, but your analysis of the problem does not prove > it. Unless there is a serious bug in framesync, it will not call > overlay_cuda_blend() without a secondary frame "due to e.g secondary > source delay.", it will only do so if the stream has not yet begun or > has ended. > > Please give more details on the exact circumstances you observe the > issue. I create a filter graph with vf_cuda_overlay in my application. I start sending frames into a "main" stream link using av_buffersrc_write_frame. I start sending frames into a "secondary" stream too almost immediately. I have one output sink in my filter graph. After each write I call av_buffersink_get_frame and get AVERROR_BUG from vf_overlay_cuda for some time initially but after several frames it starts returning mixed frame. I've checked vf_blend and other framesync filters. They do the same as I did in the fix. Maybe the problem In my application I cannot guarantee that both streams have the same timeline. Could you please point me to the place where framesync checks second stream availability before it calls it's filter(overlay_cuda_blend in my case). This topic is very complex to me now so any help from your side can really help me.
On 2020-08-21 20:46 +1000, Alex Pokotilo wrote: > On Fri, Aug 21, 2020 at 7:54 PM Nicolas George <george@nsup.org> wrote: > > > > Alex Pokotilo (12020-08-21): > > > From b3739e33fa04a9292dc6584bd2f31460aa53d478 Mon Sep 17 00:00:00 2001 > > > From: Alex Pokotilo <support@wmspanel.com> > > > Date: Fri, 21 Aug 2020 09:14:37 +0000 > > > Subject: [PATCH] libavfilter/vf_overlay_cuda: Pass the main frame to the output if secondary frame is not > > > available. now filter fails if secondary frame is not available due to e.g secondary source delay. > > > > Your Git commit message is badly formatted. The format is: > > > > context: short summary on one line > > > > After an empty line, details if necessary, on several lines and > > wrapped at a reasonable line length. > > > > I'm sorry. I'm a newbie. Am I right I just need to commit in following format: > > first line: short description...... > LF > more details here Yes, that should do the trick. Please don't forget to mention the "Why" part in the commit message body (the "more details here" part), if you re-do the patch. At the moment it would probably best to first wait for feed back on your explanation below. Alexander > Should I send a new patch as a separate email or reply to this one ? > Sorry again. > > The fix may be correct, but your analysis of the problem does not prove > > it. Unless there is a serious bug in framesync, it will not call > > overlay_cuda_blend() without a secondary frame "due to e.g secondary > > source delay.", it will only do so if the stream has not yet begun or > > has ended. > > > > Please give more details on the exact circumstances you observe the > > issue. > > I create a filter graph with vf_cuda_overlay in my application. I > start sending frames into a "main" stream link using > av_buffersrc_write_frame. > I start sending frames into a "secondary" stream too almost > immediately. I have one output sink in my filter graph. > After each write I call av_buffersink_get_frame and get AVERROR_BUG > from vf_overlay_cuda for some time initially but after several frames > it starts returning mixed frame. > I've checked vf_blend and other framesync filters. They do the same as > I did in the fix. > > Maybe the problem In my application I cannot guarantee that both > streams have the same timeline. > Could you please point me to the place where framesync checks second > stream availability before it calls it's filter(overlay_cuda_blend in > my case). > This topic is very complex to me now so any help from your side can > really help me.
Alex Pokotilo (12020-08-21): > I'm sorry. I'm a newbie. Am I right I just need to commit in following format: No need to apologize. > first line: short description...... > LF > more details here Exactly. > Should I send a new patch as a separate email or reply to this one ? It does not matter. > I create a filter graph with vf_cuda_overlay in my application. I > start sending frames into a "main" stream link using > av_buffersrc_write_frame. > I start sending frames into a "secondary" stream too almost > immediately. I have one output sink in my filter graph. What are the timestamps (and corresponding time bases) of the first frame you send on each input? > After each write I call av_buffersink_get_frame and get AVERROR_BUG > from vf_overlay_cuda for some time initially but after several frames > it starts returning mixed frame. > I've checked vf_blend and other framesync filters. They do the same as > I did in the fix. The fix is probably right, but I would like to make 100% sure, and I do not have a CUDA setup. Plus, it would help check the code of your application is right. > Maybe the problem In my application I cannot guarantee that both > streams have the same timeline. > Could you please point me to the place where framesync checks second > stream availability before it calls it's filter(overlay_cuda_blend in > my case). > This topic is very complex to me now so any help from your side can > really help me. The code in framesync is rather complex because it is designed to handle a wide variety of cases. If you are willing to make a quick test, this would be interesting: in libavfilter/framesync.c, in the function ff_framesync_init_dualinput, replace fs->in[1].before = EXT_NULL; with EXT_INFINITY and see if it changes something. But most importantly, seeing the timestamps of the frames would let me know if the issue is exactly what I think it is. Regards,
> > What are the timestamps (and corresponding time bases) of the first > frame you send on each input? this frame is sent the first fs->in[0] pts = 0; timscale=1/1000 but it doesn't matter as pts is 0 this frame is sent the second fs->in[1] pts =140 timscale=1/25 When I call av_buffersink_get_frame after sending the frame into fs->in[1] i see only the main_frame in overlay_cuda_blend function. So in my case I send the main frame with pts < pts of the second frame. > If you are willing to make a quick test, this would be interesting: in > libavfilter/framesync.c, in the function ff_framesync_init_dualinput, > replace fs->in[1].before = EXT_NULL; with EXT_INFINITY and see if it > changes something. When I change code as proposed I always see a main frame with input_overlay. I was able to find why it works after the fix checking framesync_advance function code. One observation not related to vf_overlay_cude fix if you don't mind: even with the fix you proposed if I don't send a single frame into fs->in[1](i.e secondary stream) I cannot get any output. This is because consume_from_fifos doesn't honor .before == EXT_INFINITY and .state == STATE_BOF and requires at least one frame for each input. I cannot say is it WAD or not - just my observation. When I read a description about EXT_INFINITY I guessed this flag should force processing other inputs ignoring this one. But again if I am wrong(and most probably I missed something) just ignore this.
> One observation not related to vf_overlay_cude fix if you don't mind: > even with the fix you proposed if I don't send a single frame into > fs->in[1](i.e secondary stream) I cannot get any output. > This is because consume_from_fifos doesn't honor .before == > EXT_INFINITY and .state == STATE_BOF > let me correct myself: consume_from_fifos doesn't honor .before == EXT_NULL and .state == STATE_BOF > and requires at least one frame for each input. > I cannot say is it WAD or not - just my observation. > When I read a description about EXT_INFINITY I guessed this flag > should force processing other inputs ignoring this one Again let me correct myself here too: I think that if I set fs->in[1].before== EXT_NULL (this is default behavior) framesync should return main frames even if I don't send any frames into fs->in[1]. but consume_from_fifos doesn't honor .before == EXT_NULL and .state == STATE_BOF and this is why it waits for fs->in[1] frames forever. Sorry for bothering you with unrelated question
diff --git a/libavfilter/vf_overlay_cuda.c b/libavfilter/vf_overlay_cuda.c index 2f0f860e50..355cb39019 100644 --- a/libavfilter/vf_overlay_cuda.c +++ b/libavfilter/vf_overlay_cuda.c @@ -157,8 +157,8 @@ static int overlay_cuda_blend(FFFrameSync *fs) if (ret < 0) return ret; - if (!input_main || !input_overlay) - return AVERROR_BUG; + if (!input_overlay) + return ff_filter_frame(outlink, input_main); ret = av_frame_make_writable(input_main); if (ret < 0) {