[FFmpeg-devel] fftools/ffmpeg: fix video frame lost sync for the dual input ts files with audio

Submitted by lance.lmwang@gmail.com on July 8, 2019, 10:34 a.m.

Details

Message ID 1562582052-27161-1-git-send-email-lance.lmwang@gmail.com
State New
Headers show

Commit Message

lance.lmwang@gmail.com July 8, 2019, 10:34 a.m.
From: Limin Wang <lance.lmwang@gmail.com>

How to reproduce the problem(use two ts files with audio stream):
ffmpeg -i left_w_a.ts -i right_w_a.ts -filter_complex "hstack=inputs=2" -f null -

With above command, the audio stream of the second input will be discarded default, however the start_time is 
initalized from the audio stream, so the old code will try to correct the ts offset in the condition which will
cause the video frame out of sync and report some dts error like below:
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 43 >= 43
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 44 >= 44
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 45 >= 45
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 46 >= 46
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 47 >= 47

Or you can try with below command to reproduce the issue:
ffmpeg -i input_w_a.ts -i input_w_a.ts -filter_complex "ssim=f=ssim.log" -f null -

You'll get below ssim result which isn't expected result for the two input is same, the SSIM should be 1.00000
SSIM Y:0.777353 (6.523822) U:0.964087 (14.447539) V:0.966739 (14.780661) All:0.840039 (7.959872)

With the patch applied, the above testing command can get expected result.

other solution are: use the setpts=PTS-STARTPTS filter or skip all audio stream to avoid the timestamp adjustment.

Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
---
 fftools/ffmpeg.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Comments

Michael Niedermayer July 9, 2019, 5:49 p.m.
On Mon, Jul 08, 2019 at 06:34:12PM +0800, lance.lmwang@gmail.com wrote:
> From: Limin Wang <lance.lmwang@gmail.com>
> 
> How to reproduce the problem(use two ts files with audio stream):
> ffmpeg -i left_w_a.ts -i right_w_a.ts -filter_complex "hstack=inputs=2" -f null -
> 
> With above command, the audio stream of the second input will be discarded default, however the start_time is 
> initalized from the audio stream, so the old code will try to correct the ts offset in the condition which will
> cause the video frame out of sync and report some dts error like below:
> [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 43 >= 43
> [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 44 >= 44
> [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 45 >= 45
> [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 46 >= 46
> [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 47 >= 47
> 
> Or you can try with below command to reproduce the issue:
> ffmpeg -i input_w_a.ts -i input_w_a.ts -filter_complex "ssim=f=ssim.log" -f null -
> 
> You'll get below ssim result which isn't expected result for the two input is same, the SSIM should be 1.00000
> SSIM Y:0.777353 (6.523822) U:0.964087 (14.447539) V:0.966739 (14.780661) All:0.840039 (7.959872)
> 
> With the patch applied, the above testing command can get expected result.
> 
> other solution are: use the setpts=PTS-STARTPTS filter or skip all audio stream to avoid the timestamp adjustment.
> 
> Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
> ---
>  fftools/ffmpeg.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)

this breaks fate-mpeg2-ticket186

[...]
lance.lmwang@gmail.com July 10, 2019, 2:33 p.m.
On Tue, Jul 09, 2019 at 07:49:23PM +0200, Michael Niedermayer wrote:
> On Mon, Jul 08, 2019 at 06:34:12PM +0800, lance.lmwang@gmail.com wrote:
> > From: Limin Wang <lance.lmwang@gmail.com>
> > 
> > How to reproduce the problem(use two ts files with audio stream):
> > ffmpeg -i left_w_a.ts -i right_w_a.ts -filter_complex "hstack=inputs=2" -f null -
> > 
> > With above command, the audio stream of the second input will be discarded default, however the start_time is 
> > initalized from the audio stream, so the old code will try to correct the ts offset in the condition which will
> > cause the video frame out of sync and report some dts error like below:
> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 43 >= 43
> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 44 >= 44
> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 45 >= 45
> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 46 >= 46
> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 47 >= 47
> > 
> > Or you can try with below command to reproduce the issue:
> > ffmpeg -i input_w_a.ts -i input_w_a.ts -filter_complex "ssim=f=ssim.log" -f null -
> > 
> > You'll get below ssim result which isn't expected result for the two input is same, the SSIM should be 1.00000
> > SSIM Y:0.777353 (6.523822) U:0.964087 (14.447539) V:0.966739 (14.780661) All:0.840039 (7.959872)
> > 
> > With the patch applied, the above testing command can get expected result.
> > 
> > other solution are: use the setpts=PTS-STARTPTS filter or skip all audio stream to avoid the timestamp adjustment.
> > 
> > Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
> > ---
> >  fftools/ffmpeg.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> this breaks fate-mpeg2-ticket186
> 

For the fate-mpeg2-ticket186 case, the new code wasn't triggered the timestamp
adjustment for the audio is skiped and have one stream only. So we'll get one 
extra video frame in the beginning than old code, I have checked the next
350 frames are bitexact. Can I update the tests/ref/fate/mpeg2-ticket186
to fix it?

below is the extra one frame bit info: 
+0,          0,          0,        1,   152064, 0x5c464d94 



> [...]
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> When you are offended at any man's fault, turn to yourself and study your
> own failings. Then you will forget your anger. -- Epictetus



> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Paul B Mahol July 10, 2019, 2:44 p.m.
On 7/10/19, Limin Wang <lance.lmwang@gmail.com> wrote:
> On Tue, Jul 09, 2019 at 07:49:23PM +0200, Michael Niedermayer wrote:
>> On Mon, Jul 08, 2019 at 06:34:12PM +0800, lance.lmwang@gmail.com wrote:
>> > From: Limin Wang <lance.lmwang@gmail.com>
>> >
>> > How to reproduce the problem(use two ts files with audio stream):
>> > ffmpeg -i left_w_a.ts -i right_w_a.ts -filter_complex "hstack=inputs=2"
>> > -f null -
>> >
>> > With above command, the audio stream of the second input will be
>> > discarded default, however the start_time is
>> > initalized from the audio stream, so the old code will try to correct
>> > the ts offset in the condition which will
>> > cause the video frame out of sync and report some dts error like below:
>> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically
>> > increasing dts to muxer in stream 0: 43 >= 43
>> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically
>> > increasing dts to muxer in stream 0: 44 >= 44
>> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically
>> > increasing dts to muxer in stream 0: 45 >= 45
>> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically
>> > increasing dts to muxer in stream 0: 46 >= 46
>> > [null @ 0x7fa78c000c00] Application provided invalid, non monotonically
>> > increasing dts to muxer in stream 0: 47 >= 47
>> >
>> > Or you can try with below command to reproduce the issue:
>> > ffmpeg -i input_w_a.ts -i input_w_a.ts -filter_complex "ssim=f=ssim.log"
>> > -f null -
>> >
>> > You'll get below ssim result which isn't expected result for the two
>> > input is same, the SSIM should be 1.00000
>> > SSIM Y:0.777353 (6.523822) U:0.964087 (14.447539) V:0.966739 (14.780661)
>> > All:0.840039 (7.959872)
>> >
>> > With the patch applied, the above testing command can get expected
>> > result.
>> >
>> > other solution are: use the setpts=PTS-STARTPTS filter or skip all audio
>> > stream to avoid the timestamp adjustment.
>> >
>> > Signed-off-by: Limin Wang <lance.lmwang@gmail.com>
>> > ---
>> >  fftools/ffmpeg.c | 5 ++++-
>> >  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> this breaks fate-mpeg2-ticket186
>>
>
> For the fate-mpeg2-ticket186 case, the new code wasn't triggered the
> timestamp
> adjustment for the audio is skiped and have one stream only. So we'll get
> one
> extra video frame in the beginning than old code, I have checked the next
> 350 frames are bitexact. Can I update the tests/ref/fate/mpeg2-ticket186
> to fix it?
>

Yes, if change is correct, update relevant files.

> below is the extra one frame bit info:
> +0,          0,          0,        1,   152064, 0x5c464d94
>
>
>
>> [...]
>> --
>> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>>
>> When you are offended at any man's fault, turn to yourself and study your
>> own failings. Then you will forget your anger. -- Epictetus
>
>
>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Patch hide | download patch | download mbox

diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
index 01f04103cf..3fed92639a 100644
--- a/fftools/ffmpeg.c
+++ b/fftools/ffmpeg.c
@@ -4378,13 +4378,16 @@  static int process_input(int file_index)
             && ifile->ts_offset == -is->start_time
             && (is->iformat->flags & AVFMT_TS_DISCONT)) {
             int64_t new_start_time = INT64_MAX;
+            int nb_used_streams = 0;
+
             for (i=0; i<is->nb_streams; i++) {
                 AVStream *st = is->streams[i];
                 if(st->discard == AVDISCARD_ALL || st->start_time == AV_NOPTS_VALUE)
                     continue;
+                nb_used_streams++;
                 new_start_time = FFMIN(new_start_time, av_rescale_q(st->start_time, st->time_base, AV_TIME_BASE_Q));
             }
-            if (new_start_time > is->start_time) {
+            if (nb_used_streams > 1 && new_start_time > is->start_time) {
                 av_log(is, AV_LOG_VERBOSE, "Correcting start time by %"PRId64"\n", new_start_time - is->start_time);
                 ifile->ts_offset = -new_start_time;
             }