| Message ID | 16eb1661-7d7e-f28b-ea3c-38a7b7dd8d04@gyani.pro |
|---|---|
| State | New |
| Headers | show |
On Fri, 26 Jul 2019, Gyan wrote: > > Patch supported by and tested at Google. Seems like a huge hack for a decklink issue... ;) I wonder if there are better approaches to work around this. Like dropping all frames until we have a valid signal. How is this repoducible? do you have an exact model / signal type / TC source requirement? Thanks, Marton
On 28-07-2019 01:37 AM, Marton Balint wrote: > > > On Fri, 26 Jul 2019, Gyan wrote: > >> >> Patch supported by and tested at Google. > > Seems like a huge hack for a decklink issue... ;) > > I wonder if there are better approaches to work around this. Like > dropping all frames until we have a valid signal. Let's say the scenario is like this, Frame # TC Status 1 no dropped 2 no demuxed 3 no demuxed 4 no dropped 5 04:11:21:04 dropped 6 04:11:21:05 demuxed We don't want to drop frames before 6; they still should be digitized (sometimes the TC appears after ~15 frames). But the stored TC needs to be 04:11:21:04 + (2 - 5) = 04:11:21:01 > How is this repoducible? do you have an exact model / signal type / TC > source requirement? Usually RP188 but tested over various configurations afaik, but my contact at Google has the details. I'll ask them to post. Gyan
> On 28-07-2019 01:37 AM, Marton Balint wrote: >> >> >> On Fri, 26 Jul 2019, Gyan wrote: >> >>> >>> Patch supported by and tested at Google. >> >> Seems like a huge hack for a decklink issue... ;) >> >> I wonder if there are better approaches to work around this. Like >> dropping all frames until we have a valid signal. > > Let's say the scenario is like this, > > Frame # TC Status > 1 no dropped > 2 no demuxed > 3 no demuxed > 4 no dropped > 5 04:11:21:04 dropped > 6 04:11:21:05 demuxed > > > We don't want to drop frames before 6; they still should be digitized > (sometimes the TC appears after ~15 frames). But the stored TC needs > to be 04:11:21:04 + (2 - 5) = 04:11:21:01 > >> How is this repoducible? do you have an exact model / signal type / >> TC source requirement? (Anchor for reply)
Hi Marton, The issue with the out of sync TC was reproducible on all tapes and decks that we tested. I don't have the exact number now, but a few dozens, less than 100. They all had between 7 and 17 frames out of sync. We were not able to obtain anything more in sync than 7 frames. The TC sync was tested by setting up the deck to "burn" the TC with the image while capturing the content with TC through ffmpeg. We then play the file in a player that supports timecodes and compare the burned-in TC with the one captured in the metadata. We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We tested on multiple decks from each model and confirmed the issue was present + that Gyan's patch seemed to fix it. We have used several types of Betacam tapes and HDCAM tapes. I can not comment on whether this is the best solution, but can confirm it works. Let me know if you need more info. Thanks, ilinca On Fri, Aug 9, 2019 at 1:08 PM Gyan <ffmpeg@gyani.pro> wrote: > > > On 28-07-2019 01:37 AM, Marton Balint wrote: > >> > >> > >> On Fri, 26 Jul 2019, Gyan wrote: > >> > >>> > >>> Patch supported by and tested at Google. > >> > >> Seems like a huge hack for a decklink issue... ;) > >> > >> I wonder if there are better approaches to work around this. Like > >> dropping all frames until we have a valid signal. > > > > Let's say the scenario is like this, > > > > Frame # TC Status > > 1 no dropped > > 2 no demuxed > > 3 no demuxed > > 4 no dropped > > 5 04:11:21:04 dropped > > 6 04:11:21:05 demuxed > > > > > > We don't want to drop frames before 6; they still should be digitized > > (sometimes the TC appears after ~15 frames). But the stored TC needs > > to be 04:11:21:04 + (2 - 5) = 04:11:21:01 > > > >> How is this repoducible? do you have an exact model / signal type / > >> TC source requirement? > > (Anchor for reply) > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On 09-08-2019 04:59 PM, Ilinca Tudose wrote: > Hi Marton, > > The issue with the out of sync TC was reproducible on all tapes and decks > that we tested. I don't have the exact number now, but a few dozens, less > than 100. They all had between 7 and 17 frames out of sync. We were not > able to obtain anything more in sync than 7 frames. > > The TC sync was tested by setting up the deck to "burn" the TC with the > image while capturing the content with TC through ffmpeg. We then play the > file in a player that supports timecodes and compare the burned-in TC with > the one captured in the metadata. > > We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We tested on > multiple decks from each model and confirmed the issue was present + that > Gyan's patch seemed to fix it. We have used several types of Betacam tapes > and HDCAM tapes. I can not comment on whether this is the best solution, > but can confirm it works. > > Let me know if you need more info. > > Thanks, > ilinca Ping. Gyan > > On Fri, Aug 9, 2019 at 1:08 PM Gyan <ffmpeg@gyani.pro> wrote: > >>> On 28-07-2019 01:37 AM, Marton Balint wrote: >>>> >>>> On Fri, 26 Jul 2019, Gyan wrote: >>>> >>>>> Patch supported by and tested at Google. >>>> Seems like a huge hack for a decklink issue... ;) >>>> >>>> I wonder if there are better approaches to work around this. Like >>>> dropping all frames until we have a valid signal. >>> Let's say the scenario is like this, >>> >>> Frame # TC Status >>> 1 no dropped >>> 2 no demuxed >>> 3 no demuxed >>> 4 no dropped >>> 5 04:11:21:04 dropped >>> 6 04:11:21:05 demuxed >>> >>> >>> We don't want to drop frames before 6; they still should be digitized >>> (sometimes the TC appears after ~15 frames). But the stored TC needs >>> to be 04:11:21:04 + (2 - 5) = 04:11:21:01 >>> >>>> How is this repoducible? do you have an exact model / signal type / >>>> TC source requirement? >> (Anchor for reply) >> _______________________________________________ >> ffmpeg-devel mailing list >> ffmpeg-devel@ffmpeg.org >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> >> To unsubscribe, visit link above, or email >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On Wed, 14 Aug 2019, Gyan wrote: > > > On 09-08-2019 04:59 PM, Ilinca Tudose wrote: >> Hi Marton, >> >> The issue with the out of sync TC was reproducible on all tapes and decks >> that we tested. I don't have the exact number now, but a few dozens, less >> than 100. They all had between 7 and 17 frames out of sync. We were not >> able to obtain anything more in sync than 7 frames. >> >> The TC sync was tested by setting up the deck to "burn" the TC with the >> image while capturing the content with TC through ffmpeg. We then play the >> file in a player that supports timecodes and compare the burned-in TC with >> the one captured in the metadata. >> >> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We tested on >> multiple decks from each model and confirmed the issue was present + that >> Gyan's patch seemed to fix it. We have used several types of Betacam tapes >> and HDCAM tapes. I can not comment on whether this is the best solution, >> but can confirm it works. >> >> Let me know if you need more info. >> >> Thanks, >> ilinca > > Ping. Sorry, I need a bit more time to investigate. Thanks, Marton
On Wed, 14 Aug 2019, Marton Balint wrote: > > > On Wed, 14 Aug 2019, Gyan wrote: > >> >> >> On 09-08-2019 04:59 PM, Ilinca Tudose wrote: >>> Hi Marton, >>> >>> The issue with the out of sync TC was reproducible on all tapes and decks >>> that we tested. I don't have the exact number now, but a few dozens, less >>> than 100. They all had between 7 and 17 frames out of sync. We were not >>> able to obtain anything more in sync than 7 frames. >>> >>> The TC sync was tested by setting up the deck to "burn" the TC with the >>> image while capturing the content with TC through ffmpeg. We then play the >>> file in a player that supports timecodes and compare the burned-in TC with >>> the one captured in the metadata. >>> >>> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We tested on >>> multiple decks from each model and confirmed the issue was present + that >>> Gyan's patch seemed to fix it. We have used several types of Betacam tapes >>> and HDCAM tapes. I can not comment on whether this is the best solution, >>> but can confirm it works. >>> >>> Let me know if you need more info. >>> >>> Thanks, >>> ilinca >> >> Ping. > > Sorry, I need a bit more time to investigate. OK, I did a couple of tests myself for HD and SD pal signals, and it seems to me if signal autodetection is used, so no format_code is specified then timecode is readily available in the first frame. I was using BlackMagic SDK 11.1 for testing by the way. Can you confirm the same behaviour with your setup? Because that would make the patch uneeded I believe. Thanks, Marton
On 17-08-2019 08:42 PM, Marton Balint wrote: > > > On Wed, 14 Aug 2019, Marton Balint wrote: > >> >> >> On Wed, 14 Aug 2019, Gyan wrote: >> >>> >>> >>> On 09-08-2019 04:59 PM, Ilinca Tudose wrote: >>>> Hi Marton, >>>> >>>> The issue with the out of sync TC was reproducible on all tapes and >>>> decks >>>> that we tested. I don't have the exact number now, but a few >>>> dozens, less >>>> than 100. They all had between 7 and 17 frames out of sync. We were >>>> not >>>> able to obtain anything more in sync than 7 frames. >>>> >>>> The TC sync was tested by setting up the deck to "burn" the TC with >>>> the >>>> image while capturing the content with TC through ffmpeg. We then >>>> play the >>>> file in a player that supports timecodes and compare the burned-in >>>> TC with >>>> the one captured in the metadata. >>>> >>>> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We >>>> tested on >>>> multiple decks from each model and confirmed the issue was present >>>> + that >>>> Gyan's patch seemed to fix it. We have used several types of >>>> Betacam tapes >>>> and HDCAM tapes. I can not comment on whether this is the best >>>> solution, >>>> but can confirm it works. >>>> >>>> Let me know if you need more info. >>>> >>>> Thanks, >>>> ilinca >>> >>> Ping. >> >> Sorry, I need a bit more time to investigate. > > OK, I did a couple of tests myself for HD and SD pal signals, and it > seems to me if signal autodetection is used, so no format_code is > specified then timecode is readily available in the first frame. I was > using BlackMagic SDK 11.1 for testing by the way. Can you confirm the > same behaviour with your setup? So, were you able to reproduce the issue _with_ format_code specified? Gyan
On Sat, 17 Aug 2019, Gyan wrote: > > > On 17-08-2019 08:42 PM, Marton Balint wrote: >> >> >> On Wed, 14 Aug 2019, Marton Balint wrote: >> >>> >>> >>> On Wed, 14 Aug 2019, Gyan wrote: >>> >>>> >>>> >>>> On 09-08-2019 04:59 PM, Ilinca Tudose wrote: >>>>> Hi Marton, >>>>> >>>>> The issue with the out of sync TC was reproducible on all tapes and >>>>> decks >>>>> that we tested. I don't have the exact number now, but a few >>>>> dozens, less >>>>> than 100. They all had between 7 and 17 frames out of sync. We were >>>>> not >>>>> able to obtain anything more in sync than 7 frames. >>>>> >>>>> The TC sync was tested by setting up the deck to "burn" the TC with >>>>> the >>>>> image while capturing the content with TC through ffmpeg. We then >>>>> play the >>>>> file in a player that supports timecodes and compare the burned-in >>>>> TC with >>>>> the one captured in the metadata. >>>>> >>>>> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We >>>>> tested on >>>>> multiple decks from each model and confirmed the issue was present >>>>> + that >>>>> Gyan's patch seemed to fix it. We have used several types of >>>>> Betacam tapes >>>>> and HDCAM tapes. I can not comment on whether this is the best >>>>> solution, >>>>> but can confirm it works. >>>>> >>>>> Let me know if you need more info. >>>>> >>>>> Thanks, >>>>> ilinca >>>> >>>> Ping. >>> >>> Sorry, I need a bit more time to investigate. >> >> OK, I did a couple of tests myself for HD and SD pal signals, and it >> seems to me if signal autodetection is used, so no format_code is >> specified then timecode is readily available in the first frame. I was >> using BlackMagic SDK 11.1 for testing by the way. Can you confirm the >> same behaviour with your setup? > > So, were you able to reproduce the issue _with_ format_code specified? I was able to reproduce the part where after starting the capture the decklink device records empty frames (with flag bmdFrameHasNoInputSource set) for a couple of frames. I was not able to reproduce the part where the timecode was earlier or later available then the first frame of the useful video signal. Regards, Marton
On 17-08-2019 09:21 PM, Marton Balint wrote: > > > On Sat, 17 Aug 2019, Gyan wrote: > >> >> >> On 17-08-2019 08:42 PM, Marton Balint wrote: >>> >>> >>> On Wed, 14 Aug 2019, Marton Balint wrote: >>> >>>> >>>> >>>> On Wed, 14 Aug 2019, Gyan wrote: >>>> >>>>> >>>>> >>>>> On 09-08-2019 04:59 PM, Ilinca Tudose wrote: >>>>>> Hi Marton, >>>>>> >>>>>> The issue with the out of sync TC was reproducible on all tapes >>>>>> and decks >>>>>> that we tested. I don't have the exact number now, but a few >>>>>> dozens, less >>>>>> than 100. They all had between 7 and 17 frames out of sync. We >>>>>> were not >>>>>> able to obtain anything more in sync than 7 frames. >>>>>> >>>>>> The TC sync was tested by setting up the deck to "burn" the TC >>>>>> with the >>>>>> image while capturing the content with TC through ffmpeg. We then >>>>>> play the >>>>>> file in a player that supports timecodes and compare the >>>>>> burned-in TC with >>>>>> the one captured in the metadata. >>>>>> >>>>>> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We >>>>>> tested on >>>>>> multiple decks from each model and confirmed the issue was >>>>>> present + that >>>>>> Gyan's patch seemed to fix it. We have used several types of >>>>>> Betacam tapes >>>>>> and HDCAM tapes. I can not comment on whether this is the best >>>>>> solution, >>>>>> but can confirm it works. >>>>>> >>>>>> Let me know if you need more info. >>>>>> >>>>>> Thanks, >>>>>> ilinca >>>>> >>>>> Ping. >>>> >>>> Sorry, I need a bit more time to investigate. >>> >>> OK, I did a couple of tests myself for HD and SD pal signals, and it >>> seems to me if signal autodetection is used, so no format_code is >>> specified then timecode is readily available in the first frame. I >>> was using BlackMagic SDK 11.1 for testing by the way. Can you >>> confirm the same behaviour with your setup? >> >> So, were you able to reproduce the issue _with_ format_code specified? > > I was able to reproduce the part where after starting the capture the > decklink device records empty frames (with flag > bmdFrameHasNoInputSource set) for a couple of frames. > > I was not able to reproduce the part where the timecode was earlier or > later available then the first frame of the useful video signal. Ok, thanks. I'll get back when I have more info. Gyan
Hi Marton, I want to confirm that we get correctly synced TCs when leaving out the format_code parameter and using an ffmpeg build from head (not with Gyan's patch). However, I was under the impression that we need the format_code so that we can capture different content types, in the original format. For example multiple frame rates or interlaced content. How would we set this up if we are not using format_code? Is this identified automatically? Thanks, ilinca On Sat, Aug 17, 2019 at 6:06 PM Gyan <ffmpeg@gyani.pro> wrote: > > > On 17-08-2019 09:21 PM, Marton Balint wrote: > > > > > > On Sat, 17 Aug 2019, Gyan wrote: > > > >> > >> > >> On 17-08-2019 08:42 PM, Marton Balint wrote: > >>> > >>> > >>> On Wed, 14 Aug 2019, Marton Balint wrote: > >>> > >>>> > >>>> > >>>> On Wed, 14 Aug 2019, Gyan wrote: > >>>> > >>>>> > >>>>> > >>>>> On 09-08-2019 04:59 PM, Ilinca Tudose wrote: > >>>>>> Hi Marton, > >>>>>> > >>>>>> The issue with the out of sync TC was reproducible on all tapes > >>>>>> and decks > >>>>>> that we tested. I don't have the exact number now, but a few > >>>>>> dozens, less > >>>>>> than 100. They all had between 7 and 17 frames out of sync. We > >>>>>> were not > >>>>>> able to obtain anything more in sync than 7 frames. > >>>>>> > >>>>>> The TC sync was tested by setting up the deck to "burn" the TC > >>>>>> with the > >>>>>> image while capturing the content with TC through ffmpeg. We then > >>>>>> play the > >>>>>> file in a player that supports timecodes and compare the > >>>>>> burned-in TC with > >>>>>> the one captured in the metadata. > >>>>>> > >>>>>> We used Decklink quad 2 and several Sony decks: J30, J3, JH3. We > >>>>>> tested on > >>>>>> multiple decks from each model and confirmed the issue was > >>>>>> present + that > >>>>>> Gyan's patch seemed to fix it. We have used several types of > >>>>>> Betacam tapes > >>>>>> and HDCAM tapes. I can not comment on whether this is the best > >>>>>> solution, > >>>>>> but can confirm it works. > >>>>>> > >>>>>> Let me know if you need more info. > >>>>>> > >>>>>> Thanks, > >>>>>> ilinca > >>>>> > >>>>> Ping. > >>>> > >>>> Sorry, I need a bit more time to investigate. > >>> > >>> OK, I did a couple of tests myself for HD and SD pal signals, and it > >>> seems to me if signal autodetection is used, so no format_code is > >>> specified then timecode is readily available in the first frame. I > >>> was using BlackMagic SDK 11.1 for testing by the way. Can you > >>> confirm the same behaviour with your setup? > >> > >> So, were you able to reproduce the issue _with_ format_code specified? > > > > I was able to reproduce the part where after starting the capture the > > decklink device records empty frames (with flag > > bmdFrameHasNoInputSource set) for a couple of frames. > > > > I was not able to reproduce the part where the timecode was earlier or > > later available then the first frame of the useful video signal. > > Ok, thanks. I'll get back when I have more info. > > Gyan > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
On 19-08-2019 07:15 PM, Ilinca Tudose wrote: > Hi Marton, > > I want to confirm that we get correctly synced TCs when leaving out the > format_code parameter and using an ffmpeg build from head (not with Gyan's > patch). However, I was under the impression that we need the format_code so > that we can capture different content types, in the original format. For > example multiple frame rates or interlaced content. How would we set this > up if we are not using format_code? Is this identified automatically? A couple of follow-up Qs: Is auto-detection available for all Decklink devices? For those for which it is available, are there any edge cases in which it sets inaccurate mode? Gyan
> A couple of follow-up Qs: > > Is auto-detection available for all Decklink devices? No, but AFAIK it is for all devices which support SDI. Generally it's the older analog capture devices which don't support it. > For those for which it is available, are there any edge cases in which > it sets inaccurate mode? I don't trust the existing detection code enough to use it in production. It often fails to detect and thus ffmpeg will exit at startup. Also, there are cases where it will misdetect 1080i59 as 1080p30 depending on the card. It's been on my TODO list for a while to make that code more robust (I believe I know what most of the issues are), but it hasn't been critical for any of my use cases. Devin
On Tue, 20 Aug 2019, Devin Heitmueller wrote: >> A couple of follow-up Qs: >> >> Is auto-detection available for all Decklink devices? > > No, but AFAIK it is for all devices which support SDI. Generally it's > the older analog capture devices which don't support it. > >> For those for which it is available, are there any edge cases in which >> it sets inaccurate mode? > > I don't trust the existing detection code enough to use it in > production. It often fails to detect and thus ffmpeg will exit at > startup. Can this be fixed by simply increasing the timeout from 1 sec to 2 seconds? > Also, there are cases where it will misdetect 1080i59 as > 1080p30 depending on the card. It's been on my TODO list for a while > to make that code more robust (I believe I know what most of the > issues are), but it hasn't been critical for any of my use cases. Hmm, interesting... When you say the issue is card-dependant, does this mean card _model_ dependant or that the issue can affect one card and not the other with of the same model/fw? Thanks, Marton
On 22-08-2019 01:10 AM, Marton Balint wrote: > > > On Tue, 20 Aug 2019, Devin Heitmueller wrote: > >>> A couple of follow-up Qs: >>> >>> Is auto-detection available for all Decklink devices? >> >> No, but AFAIK it is for all devices which support SDI. Generally it's >> the older analog capture devices which don't support it. >> >>> For those for which it is available, are there any edge cases in which >>> it sets inaccurate mode? >> >> I don't trust the existing detection code enough to use it in >> production. It often fails to detect and thus ffmpeg will exit at >> startup. > > Can this be fixed by simply increasing the timeout from 1 sec to 2 > seconds? > >> Also, there are cases where it will misdetect 1080i59 as >> 1080p30 depending on the card. It's been on my TODO list for a while >> to make that code more robust (I believe I know what most of the >> issues are), but it hasn't been critical for any of my use cases. > > Hmm, interesting... When you say the issue is card-dependant, does > this mean card _model_ dependant or that the issue can affect one card > and not the other with of the same model/fw? If an user can't or won't use format auto detection, then there is no way to reliably obtain the correct timecode without this patch or some similar hack. Maybe the offset-related ops could be guarded by a check for auto detection, even though the patch won't hurt even if it was used in that case. Gyan
On Fri, 30 Aug 2019, Gyan wrote: > > > On 22-08-2019 01:10 AM, Marton Balint wrote: >> >> >> On Tue, 20 Aug 2019, Devin Heitmueller wrote: >> >>>> A couple of follow-up Qs: >>>> >>>> Is auto-detection available for all Decklink devices? >>> >>> No, but AFAIK it is for all devices which support SDI. Generally it's >>> the older analog capture devices which don't support it. >>> >>>> For those for which it is available, are there any edge cases in which >>>> it sets inaccurate mode? >>> >>> I don't trust the existing detection code enough to use it in >>> production. It often fails to detect and thus ffmpeg will exit at >>> startup. >> >> Can this be fixed by simply increasing the timeout from 1 sec to 2 >> seconds? >> >>> Also, there are cases where it will misdetect 1080i59 as >>> 1080p30 depending on the card. It's been on my TODO list for a while >>> to make that code more robust (I believe I know what most of the >>> issues are), but it hasn't been critical for any of my use cases. >> >> Hmm, interesting... When you say the issue is card-dependant, does >> this mean card _model_ dependant or that the issue can affect one card >> and not the other with of the same model/fw? > > If an user can't or won't use format auto detection, then there is no > way to reliably obtain the correct timecode without this patch or some > similar hack. > > Maybe the offset-related ops could be guarded by a check for auto > detection, even though the patch won't hurt even if it was used in that > case. I am still not liking this too much because we are guessing some timecode which we never actually saw, and the guessing is based on the number of the VideoFrameArrived callbacks. Yes, usually it is called once for every frame, but some frames might not contain audio, also there are some examples in the SDK docs where only an audio frame (and no video frame) is available in the callback. So it is up to the receiving application to do something to keep audio and video in sync, it either drops/duplicates video or audio. If it drops/dups video then our guessed start timecode can be wrong even if we were only counting the video frames. So I think that if a workaround/hack is needed, we should do that differently. One idea is to introduce an option which makes the code drop video till the first valid video frame, or the first valid timecode. But if autodetection works (or it can be fixed), then I just don't see too much use even for this. Regards, Marton
On 31-08-2019 04:14 AM, Marton Balint wrote: > > > On Fri, 30 Aug 2019, Gyan wrote: > >> >> >> On 22-08-2019 01:10 AM, Marton Balint wrote: >>> >>> >>> On Tue, 20 Aug 2019, Devin Heitmueller wrote: >>> >>>>> A couple of follow-up Qs: >>>>> >>>>> Is auto-detection available for all Decklink devices? >>>> >>>> No, but AFAIK it is for all devices which support SDI. Generally it's >>>> the older analog capture devices which don't support it. >>>> >>>>> For those for which it is available, are there any edge cases in >>>>> which >>>>> it sets inaccurate mode? >>>> >>>> I don't trust the existing detection code enough to use it in >>>> production. It often fails to detect and thus ffmpeg will exit at >>>> startup. >>> >>> Can this be fixed by simply increasing the timeout from 1 sec to 2 >>> seconds? >>> >>>> Also, there are cases where it will misdetect 1080i59 as >>>> 1080p30 depending on the card. It's been on my TODO list for a while >>>> to make that code more robust (I believe I know what most of the >>>> issues are), but it hasn't been critical for any of my use cases. >>> >>> Hmm, interesting... When you say the issue is card-dependant, does >>> this mean card _model_ dependant or that the issue can affect one >>> card and not the other with of the same model/fw? >> >> If an user can't or won't use format auto detection, then there is no >> way to reliably obtain the correct timecode without this patch or >> some similar hack. >> >> Maybe the offset-related ops could be guarded by a check for auto >> detection, even though the patch won't hurt even if it was used in >> that case. > > I am still not liking this too much because we are guessing some > timecode which we never actually saw, and the guessing is based on the > number of the VideoFrameArrived callbacks. Yes, usually it is called > once for every frame, but some frames might not contain audio, also > there are some examples in the SDK docs where only an audio frame (and > no video frame) is available in the callback. Yes, that's why there is a new counter, incremented only upon getting a video frame and reset until the demuxer successfully queues the packet of the first video frame. I just noticed an issue with it, but I can change that. > So it is up to the receiving application to do something to keep audio > and video in sync, it either drops/duplicates video or audio. If it > drops/dups video then our guessed start timecode can be wrong even if > we were only counting the video frames. By receiving app, do you mean the Decklink driver or decklink_dec/ffmpeg.c? > So I think that if a workaround/hack is needed, we should do that > differently. One idea is to introduce an option which makes the code > drop video till the first valid video frame, or the first valid > timecode. But if autodetection works (or it can be fixed), then I just > don't see too much use even for this. As Ilinca's tests showed, without auto format detection, the stored timecode is *wrong*. That's a bug so some change is *required*. Gyan
On Sat, 31 Aug 2019, Gyan wrote: > > > On 31-08-2019 04:14 AM, Marton Balint wrote: >> >> >> On Fri, 30 Aug 2019, Gyan wrote: >> >>> >>> >>> On 22-08-2019 01:10 AM, Marton Balint wrote: >>>> >>>> >>>> On Tue, 20 Aug 2019, Devin Heitmueller wrote: >>>> >>>>>> A couple of follow-up Qs: >>>>>> >>>>>> Is auto-detection available for all Decklink devices? >>>>> >>>>> No, but AFAIK it is for all devices which support SDI. Generally it's >>>>> the older analog capture devices which don't support it. >>>>> >>>>>> For those for which it is available, are there any edge cases in >>>>>> which >>>>>> it sets inaccurate mode? >>>>> >>>>> I don't trust the existing detection code enough to use it in >>>>> production. It often fails to detect and thus ffmpeg will exit at >>>>> startup. >>>> >>>> Can this be fixed by simply increasing the timeout from 1 sec to 2 >>>> seconds? >>>> >>>>> Also, there are cases where it will misdetect 1080i59 as >>>>> 1080p30 depending on the card. It's been on my TODO list for a while >>>>> to make that code more robust (I believe I know what most of the >>>>> issues are), but it hasn't been critical for any of my use cases. >>>> >>>> Hmm, interesting... When you say the issue is card-dependant, does >>>> this mean card _model_ dependant or that the issue can affect one >>>> card and not the other with of the same model/fw? >>> >>> If an user can't or won't use format auto detection, then there is no >>> way to reliably obtain the correct timecode without this patch or >>> some similar hack. >>> >>> Maybe the offset-related ops could be guarded by a check for auto >>> detection, even though the patch won't hurt even if it was used in >>> that case. >> >> I am still not liking this too much because we are guessing some >> timecode which we never actually saw, and the guessing is based on the >> number of the VideoFrameArrived callbacks. Yes, usually it is called >> once for every frame, but some frames might not contain audio, also >> there are some examples in the SDK docs where only an audio frame (and >> no video frame) is available in the callback. > > Yes, that's why there is a new counter, incremented only upon getting a > video frame and reset until the demuxer successfully queues the packet > of the first video frame. I just noticed an issue with it, but I can > change that. > > >> So it is up to the receiving application to do something to keep audio >> and video in sync, it either drops/duplicates video or audio. If it >> drops/dups video then our guessed start timecode can be wrong even if >> we were only counting the video frames. > > By receiving app, do you mean the Decklink driver or > decklink_dec/ffmpeg.c? ffmpeg.c > > >> So I think that if a workaround/hack is needed, we should do that >> differently. One idea is to introduce an option which makes the code >> drop video till the first valid video frame, or the first valid >> timecode. But if autodetection works (or it can be fixed), then I just >> don't see too much use even for this. > > As Ilinca's tests showed, without auto format detection, the stored > timecode is *wrong*. That's a bug so some change is *required*. The bug is that decklink_dec sets a timecode to AVStream metadata which is not the timecode of the first frame. As far as I recall when accepting the decklink timecode support patch, I pointed out that it is not very useful because it might not be the TC of the first frame. I am fine with a patch which only sets AVStream timecode if it is indeed the first frame. I am fine with a patch which drops starter frames without timecode when some option is set. I am not fine with a patch which does calculations in deckink_dec to guess a starter timecode. Regards, Marton
On 31-08-2019 08:11 PM, Marton Balint wrote: > > > On Sat, 31 Aug 2019, Gyan wrote: > >> >> >> On 31-08-2019 04:14 AM, Marton Balint wrote: >>> >>> >>> On Fri, 30 Aug 2019, Gyan wrote: >>> >>>> >>>> >>>> On 22-08-2019 01:10 AM, Marton Balint wrote: >>>>> >>>>> >>>>> On Tue, 20 Aug 2019, Devin Heitmueller wrote: >>>>> >>>>>>> A couple of follow-up Qs: >>>>>>> >>>>>>> Is auto-detection available for all Decklink devices? >>>>>> >>>>>> No, but AFAIK it is for all devices which support SDI. Generally >>>>>> it's >>>>>> the older analog capture devices which don't support it. >>>>>> >>>>>>> For those for which it is available, are there any edge cases in >>>>>>> which >>>>>>> it sets inaccurate mode? >>>>>> >>>>>> I don't trust the existing detection code enough to use it in >>>>>> production. It often fails to detect and thus ffmpeg will exit at >>>>>> startup. >>>>> >>>>> Can this be fixed by simply increasing the timeout from 1 sec to 2 >>>>> seconds? >>>>> >>>>>> Also, there are cases where it will misdetect 1080i59 as >>>>>> 1080p30 depending on the card. It's been on my TODO list for a >>>>>> while >>>>>> to make that code more robust (I believe I know what most of the >>>>>> issues are), but it hasn't been critical for any of my use cases. >>>>> >>>>> Hmm, interesting... When you say the issue is card-dependant, does >>>>> this mean card _model_ dependant or that the issue can affect one >>>>> card and not the other with of the same model/fw? >>>> >>>> If an user can't or won't use format auto detection, then there is >>>> no way to reliably obtain the correct timecode without this patch >>>> or some similar hack. >>>> >>>> Maybe the offset-related ops could be guarded by a check for auto >>>> detection, even though the patch won't hurt even if it was used in >>>> that case. >>> >>> I am still not liking this too much because we are guessing some >>> timecode which we never actually saw, and the guessing is based on >>> the number of the VideoFrameArrived callbacks. Yes, usually it is >>> called once for every frame, but some frames might not contain >>> audio, also there are some examples in the SDK docs where only an >>> audio frame (and no video frame) is available in the callback. >> >> Yes, that's why there is a new counter, incremented only upon getting >> a video frame and reset until the demuxer successfully queues the >> packet of the first video frame. I just noticed an issue with it, but >> I can change that. >> >> >>> So it is up to the receiving application to do something to keep >>> audio and video in sync, it either drops/duplicates video or audio. >>> If it drops/dups video then our guessed start timecode can be wrong >>> even if we were only counting the video frames. >> >> By receiving app, do you mean the Decklink driver or >> decklink_dec/ffmpeg.c? > > ffmpeg.c > >> >> >>> So I think that if a workaround/hack is needed, we should do that >>> differently. One idea is to introduce an option which makes the code >>> drop video till the first valid video frame, or the first valid >>> timecode. But if autodetection works (or it can be fixed), then I >>> just don't see too much use even for this. >> >> As Ilinca's tests showed, without auto format detection, the stored >> timecode is *wrong*. That's a bug so some change is *required*. > > The bug is that decklink_dec sets a timecode to AVStream metadata > which is not the timecode of the first frame. As far as I recall when > accepting the decklink timecode support patch, I pointed out that it > is not very useful because it might not be the TC of the first frame. > > I am fine with a patch which only sets AVStream timecode if it is > indeed the first frame. I am fine with a patch which drops starter > frames without timecode when some option is set. I am not fine with a > patch which does calculations in deckink_dec to guess a starter timecode. Ok. Will get back. Thanks, Gyan
diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h index 921818ba41..f149e7ff66 100644 --- a/libavdevice/decklink_common.h +++ b/libavdevice/decklink_common.h @@ -149,6 +149,10 @@ struct decklink_ctx { int channels; int audio_depth; + + /* Fields for timecode correction */ + unsigned long vidframeCount; // frameCount tracks audio-only packets as well + unsigned long firstframeIndex; // value of frameCount for first successfully demuxed video frame }; typedef enum { DIRECTION_IN, DIRECTION_OUT} decklink_direction_t; diff --git a/libavdevice/decklink_dec.cpp b/libavdevice/decklink_dec.cpp index 4da9122bff..d0ebd0ac0a 100644 --- a/libavdevice/decklink_dec.cpp +++ b/libavdevice/decklink_dec.cpp @@ -41,6 +41,7 @@ extern "C" { #include "libavutil/imgutils.h" #include "libavutil/intreadwrite.h" #include "libavutil/time.h" +#include "libavutil/timecode.h" #include "libavutil/mathematics.h" #include "libavutil/reverse.h" #include "avdevice.h" @@ -736,6 +737,8 @@ HRESULT decklink_input_callback::VideoInputFrameArrived( videoFrame->GetStreamTime(&frameTime, &frameDuration, ctx->video_st->time_base.den); + ctx->vidframeCount++; + if (videoFrame->GetFlags() & bmdFrameHasNoInputSource) { if (ctx->draw_bars && videoFrame->GetPixelFormat() == bmdFormat8BitYUV) { unsigned bars[8] = { @@ -765,6 +768,8 @@ HRESULT decklink_input_callback::VideoInputFrameArrived( // Handle Timecode (if requested) if (ctx->tc_format) { + if (!ctx->firstframeIndex) + ctx->firstframeIndex = ctx->vidframeCount; IDeckLinkTimecode *timecode; if (videoFrame->GetTimecode(ctx->tc_format, &timecode) == S_OK) { const char *tc = NULL; @@ -778,7 +783,9 @@ HRESULT decklink_input_callback::VideoInputFrameArrived( AVDictionary* metadata_dict = NULL; int metadata_len; uint8_t* packed_metadata; + unsigned long offset = ctx->firstframeIndex - ctx->vidframeCount; // represents offset of first returned frame relative to first frame with timecode if (av_dict_set(&metadata_dict, "timecode", tc, AV_DICT_DONT_STRDUP_VAL) >= 0) { + av_dict_set_int(&metadata_dict, "tc_offset", offset, 0); packed_metadata = av_packet_pack_dictionary(metadata_dict, &metadata_len); av_dict_free(&metadata_dict); if (packed_metadata) { @@ -880,6 +887,8 @@ HRESULT decklink_input_callback::VideoInputFrameArrived( videoFrame->AddRef(); if (avpacket_queue_put(&ctx->queue, &pkt) < 0) { + if (ctx->firstframeIndex == ctx->vidframeCount) + ctx->firstframeIndex = 0; ++ctx->dropped; } } @@ -1266,10 +1275,48 @@ int ff_decklink_read_packet(AVFormatContext *avctx, AVPacket *pkt) if (ctx->tc_format && !(av_dict_get(ctx->video_st->metadata, "timecode", NULL, 0))) { int size; const uint8_t *side_metadata = av_packet_get_side_data(pkt, AV_PKT_DATA_STRINGS_METADATA, &size); + if (side_metadata) { - if (av_packet_unpack_dictionary(side_metadata, size, &ctx->video_st->metadata) < 0) - av_log(avctx, AV_LOG_ERROR, "Unable to set timecode\n"); - } + char offset_tc[AV_TIMECODE_STR_SIZE]; + int ret; + AVTimecode tc; + AVRational vid_rate = ctx->video_st->r_frame_rate; + int64_t offset = 0; + AVDictionary *first_tc = NULL; + AVDictionaryEntry *tcstr = NULL; + AVDictionaryEntry *offsetstr = NULL; + + if (av_packet_unpack_dictionary(side_metadata, size, &first_tc) < 0 || + !(tcstr = av_dict_get(first_tc, "timecode", NULL, 0)) ) { + av_log(avctx, AV_LOG_ERROR, "Unable to find timecode\n"); + return 0; + } + + if (tcstr) + av_log(avctx, AV_LOG_DEBUG, "Serial frame timecode is %s.\n", tcstr->value); + + if (av_timecode_init_from_string(&tc, vid_rate, tcstr->value, avctx) < 0) { + av_log(avctx, AV_LOG_ERROR, "Unable to set timecode\n"); + return 0; + } + + offsetstr = av_dict_get(first_tc, "tc_offset", NULL, 0); + if (offsetstr) { + offset = strtol(offsetstr->value, NULL, 10); + av_log(avctx, AV_LOG_INFO, "Timecode offset is %" PRId64 ".\n", offset); + } else + av_log(avctx, AV_LOG_WARNING, "Unable to get correction offset for timecode.\n"); + + ret = av_dict_set(&ctx->video_st->metadata, "timecode", av_timecode_make_string(&tc, offset_tc, offset), 0); + if (ret >= 0) + av_log(avctx, AV_LOG_INFO, "Timecode %s after offset of %" PRId64 " is stored.\n", offset_tc, offset); + else + av_log(avctx, AV_LOG_ERROR, "Unable to store timecode.\n"); + + return 0; + + } else + av_log(avctx, AV_LOG_TRACE, "No timecode side data found as of pkt pts %" PRId64 "\n", pkt->pts); } return 0;
Patch supported by and tested at Google. Gyan From e51bd8201ddf618dce33ada70a9bc6ce2f33b07b Mon Sep 17 00:00:00 2001 From: Gyan Doshi <ffmpeg@gyani.pro> Date: Mon, 1 Jul 2019 23:43:44 +0530 Subject: [PATCH] avdevice/decklink: adjust for timecode lag The decklink demuxer may not start receiving timecode with the first frame, but only some frames later. The demuxer, at present, naively stores the first received timecode. This patch monitors the lag in video frames, and adjusts the timecode before saving it in stream metadata. --- libavdevice/decklink_common.h | 4 +++ libavdevice/decklink_dec.cpp | 53 +++++++++++++++++++++++++++++++++-- 2 files changed, 54 insertions(+), 3 deletions(-)