[FFmpeg-devel] (for discussion): nvenc: fix wrong aspect ratio for 720x576 and 720x480 resolution

Submitted by Miroslav Slugeň on Feb. 12, 2017, 9:25 p.m.

Details

Message ID 58A0D2BD.2020108@email.cz
State New
Headers show

Commit Message

Miroslav Slugeň Feb. 12, 2017, 9:25 p.m.
This patch is for discussion only, not ready to commit yet and maybe 
newer will be.

NVENC in current CUDA 8 SDK is setting wrong aspect ratio when encoding 
resolution 720x576 and 720x480 and using AR 4:3 or 16:9 it will encode 
to h264 header that 15:11 and 20:11 is used. This is very very very ugly 
way how to fix it. I hope someone in NVIDIA will fix this soon, because 
all encoded streams are not displayed correctly for example in videojs 
player.

Support for HEVC is currently broken.

Comments

Timo Rothenpieler Feb. 12, 2017, 9:31 p.m.
On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:
> This patch is for discussion only, not ready to commit yet and maybe
> newer will be.
> 
> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when encoding
> resolution 720x576 and 720x480 and using AR 4:3 or 16:9 it will encode
> to h264 header that 15:11 and 20:11 is used. This is very very very ugly
> way how to fix it. I hope someone in NVIDIA will fix this soon, because
> all encoded streams are not displayed correctly for example in videojs
> player.

nvenc.c had some compensation for this, which was somewhat recently
removed, because nvidia fixed something regarding it:

http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
Miroslav Slugeň Feb. 12, 2017, 9:43 p.m.
Dne 12.2.2017 v 22:31 Timo Rothenpieler napsal(a):
> On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:
>> This patch is for discussion only, not ready to commit yet and maybe
>> newer will be.
>>
>> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when encoding
>> resolution 720x576 and 720x480 and using AR 4:3 or 16:9 it will encode
>> to h264 header that 15:11 and 20:11 is used. This is very very very ugly
>> way how to fix it. I hope someone in NVIDIA will fix this soon, because
>> all encoded streams are not displayed correctly for example in videojs
>> player.
> nvenc.c had some compensation for this, which was somewhat recently
> removed, because nvidia fixed something regarding it:
>
> http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
I know that, i am monitoring all nvenc and cuvid changes in ffmpeg.

Fix is not working anymore in current drivers. What is working is if you 
set aspect ratio 16000x9001 it will give you almost correct output, but 
fixing this in h264 SPP is better solution.

M.
Timo Rothenpieler Feb. 12, 2017, 10:15 p.m.
On 2/12/2017 10:43 PM, Miroslav Slugeň wrote:
> Dne 12.2.2017 v 22:31 Timo Rothenpieler napsal(a):
>> On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:
>>> This patch is for discussion only, not ready to commit yet and maybe
>>> newer will be.
>>>
>>> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when encoding
>>> resolution 720x576 and 720x480 and using AR 4:3 or 16:9 it will encode
>>> to h264 header that 15:11 and 20:11 is used. This is very very very ugly
>>> way how to fix it. I hope someone in NVIDIA will fix this soon, because
>>> all encoded streams are not displayed correctly for example in videojs
>>> player.
>> nvenc.c had some compensation for this, which was somewhat recently
>> removed, because nvidia fixed something regarding it:
>>
>> http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
> I know that, i am monitoring all nvenc and cuvid changes in ffmpeg.
> 
> Fix is not working anymore in current drivers. What is working is if you
> set aspect ratio 16000x9001 it will give you almost correct output, but
> fixing this in h264 SPP is better solution.
> 
> M.

Imo the best would be to contact nvidia about this potential bug.
Maybe a leftover from removing the old compensation logic?
Actually fixing the driver would definitely be the best solution here.

Adding some of the @nvidia.com people from this ML to CC, maybe they can
forward the issue to the right place.
Philip Langdale Feb. 12, 2017, 10:20 p.m.
On Sun, 12 Feb 2017 22:43:41 +0100
Miroslav Slugeň <thunder.m@email.cz> wrote:

> Dne 12.2.2017 v 22:31 Timo Rothenpieler napsal(a):
> > On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:  
> >> This patch is for discussion only, not ready to commit yet and
> >> maybe newer will be.
> >>
> >> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when
> >> encoding resolution 720x576 and 720x480 and using AR 4:3 or 16:9
> >> it will encode to h264 header that 15:11 and 20:11 is used. This
> >> is very very very ugly way how to fix it. I hope someone in NVIDIA
> >> will fix this soon, because all encoded streams are not displayed
> >> correctly for example in videojs player.  
> > nvenc.c had some compensation for this, which was somewhat recently
> > removed, because nvidia fixed something regarding it:
> >
> > http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel  
> I know that, i am monitoring all nvenc and cuvid changes in ffmpeg.
> 
> Fix is not working anymore in current drivers. What is working is if
> you set aspect ratio 16000x9001 it will give you almost correct
> output, but fixing this in h264 SPP is better solution.
> 
> M.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

If it's still happening, let's just restore the original workaround.
But I stopped being able to reproduce the problem locally, and I can't
reproduce it right now with a 720x576 sample.

Note that the SDK is completely irrelevant to playback behaviour. The
CUDA SDK doesn't include current cuvid/nvenc headers and the Video SDK
isn't required because we bundle/reimplement headers with ffmpeg.

You are likely not using a new enough driver release to have the fix in
it. I'm using 378.09 here.

--phil
Miroslav Slugeň Feb. 13, 2017, 7:52 a.m.
Dne 12.2.2017 v 23:20 Philip Langdale napsal(a):
> On Sun, 12 Feb 2017 22:43:41 +0100
> Miroslav Slugeň <thunder.m@email.cz> wrote:
>
>> Dne 12.2.2017 v 22:31 Timo Rothenpieler napsal(a):
>>> On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:
>>>> This patch is for discussion only, not ready to commit yet and
>>>> maybe newer will be.
>>>>
>>>> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when
>>>> encoding resolution 720x576 and 720x480 and using AR 4:3 or 16:9
>>>> it will encode to h264 header that 15:11 and 20:11 is used. This
>>>> is very very very ugly way how to fix it. I hope someone in NVIDIA
>>>> will fix this soon, because all encoded streams are not displayed
>>>> correctly for example in videojs player.
>>> nvenc.c had some compensation for this, which was somewhat recently
>>> removed, because nvidia fixed something regarding it:
>>>
>>> http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> I know that, i am monitoring all nvenc and cuvid changes in ffmpeg.
>>
>> Fix is not working anymore in current drivers. What is working is if
>> you set aspect ratio 16000x9001 it will give you almost correct
>> output, but fixing this in h264 SPP is better solution.
>>
>> M.
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> If it's still happening, let's just restore the original workaround.
> But I stopped being able to reproduce the problem locally, and I can't
> reproduce it right now with a 720x576 sample.
>
> Note that the SDK is completely irrelevant to playback behaviour. The
> CUDA SDK doesn't include current cuvid/nvenc headers and the Video SDK
> isn't required because we bundle/reimplement headers with ffmpeg.
>
> You are likely not using a new enough driver release to have the fix in
> it. I'm using 378.09 here.
>
> --phil
>
I am using current STABLE drivers 375.26, because BETA drivers 378.09 
caused some crashes while encoding on NVENC.

I tested this on BETA drivers too and it is still same.

Original workaround is not working anymore :(

INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main) ([2][0][0][0] / 
0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 
25 tbr, 90k tbn, 50 tbc

OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), 
yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25 tbr, 90k 
tbn, 50 tbc

COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid -i 
"in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f mpegts "out.ts"

Also someone else is complaining about this issue: 
http://superuser.com/questions/1174097/ffmpegnvenc-encoding-strange-aspect-ratio

M.
Philip Langdale Feb. 13, 2017, 3:21 p.m.
On Mon, 13 Feb 2017 08:52:34 +0100
Miroslav Slugeň <thunder.m@email.cz> wrote:

> Dne 12.2.2017 v 23:20 Philip Langdale napsal(a):
> > On Sun, 12 Feb 2017 22:43:41 +0100
> > Miroslav Slugeň <thunder.m@email.cz> wrote:
> >  
> >> Dne 12.2.2017 v 22:31 Timo Rothenpieler napsal(a):  
> >>> On 2/12/2017 10:25 PM, Miroslav Slugeň wrote:  
> >>>> This patch is for discussion only, not ready to commit yet and
> >>>> maybe newer will be.
> >>>>
> >>>> NVENC in current CUDA 8 SDK is setting wrong aspect ratio when
> >>>> encoding resolution 720x576 and 720x480 and using AR 4:3 or 16:9
> >>>> it will encode to h264 header that 15:11 and 20:11 is used. This
> >>>> is very very very ugly way how to fix it. I hope someone in
> >>>> NVIDIA will fix this soon, because all encoded streams are not
> >>>> displayed correctly for example in videojs player.  
> >>> nvenc.c had some compensation for this, which was somewhat
> >>> recently removed, because nvidia fixed something regarding it:
> >>>
> >>> http://git.videolan.org/?p=ffmpeg.git;a=commit;h=829db8effd76b579ae9aca5ee8f85d3ade6af253
> >>> _______________________________________________
> >>> ffmpeg-devel mailing list
> >>> ffmpeg-devel@ffmpeg.org
> >>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel  
> >> I know that, i am monitoring all nvenc and cuvid changes in ffmpeg.
> >>
> >> Fix is not working anymore in current drivers. What is working is
> >> if you set aspect ratio 16000x9001 it will give you almost correct
> >> output, but fixing this in h264 SPP is better solution.
> >>
> >> M.
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel@ffmpeg.org
> >> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel  
> > If it's still happening, let's just restore the original workaround.
> > But I stopped being able to reproduce the problem locally, and I
> > can't reproduce it right now with a 720x576 sample.
> >
> > Note that the SDK is completely irrelevant to playback behaviour.
> > The CUDA SDK doesn't include current cuvid/nvenc headers and the
> > Video SDK isn't required because we bundle/reimplement headers with
> > ffmpeg.
> >
> > You are likely not using a new enough driver release to have the
> > fix in it. I'm using 378.09 here.
> >
> > --phil
> >  
> I am using current STABLE drivers 375.26, because BETA drivers 378.09 
> caused some crashes while encoding on NVENC.
> 
> I tested this on BETA drivers too and it is still same.
> 
> Original workaround is not working anymore :(
> 
> INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main) ([2][0][0][0] / 
> 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25
> fps, 25 tbr, 90k tbn, 50 tbc
> 
> OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] /
> 0x001B), yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25 fps,
> 25 tbr, 90k tbn, 50 tbc
> 
> COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid -i 
> "in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f mpegts
> "out.ts"
> 
> Also someone else is complaining about this issue: 
> http://superuser.com/questions/1174097/ffmpegnvenc-encoding-strange-aspect-ratio
> 
> M.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Can you point me to a sample that you see this behaviour with? I cannot
reproduce with DVD sources here, which is where I saw the original
problem.

--phil
Philip Langdale Feb. 13, 2017, 4:07 p.m.
On Mon, 13 Feb 2017 07:21:51 -0800
Philip Langdale <philipl@overt.org> wrote:

> On Mon, 13 Feb 2017 08:52:34 +0100
> Miroslav Slugeň <thunder.m@email.cz> wrote:
> 
> > >    
> > I am using current STABLE drivers 375.26, because BETA drivers
> > 378.09 caused some crashes while encoding on NVENC.
> > 
> > I tested this on BETA drivers too and it is still same.
> > 
> > Original workaround is not working anymore :(
> > 
> > INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main) ([2][0][0][0] / 
> > 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25
> > fps, 25 tbr, 90k tbn, 50 tbc
> > 
> > OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] /
> > 0x001B), yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25
> > fps, 25 tbr, 90k tbn, 50 tbc
> > 
> > COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid -i 
> > "in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f mpegts
> > "out.ts"
> > 
> > Also someone else is complaining about this issue: 
> > http://superuser.com/questions/1174097/ffmpegnvenc-encoding-strange-aspect-ratio
> > 
> > M.
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel  
> 
> Can you point me to a sample that you see this behaviour with? I
> cannot reproduce with DVD sources here, which is where I saw the
> original problem.

You sent me a sample and I tried it out. I was able to reproduce your
problem, but it is not the original problem, and I wonder what is
really going on.

If you take your sample and your command line and then output to a
different container (i tried mkv and mp4), it's all correct - and it
would not be correct if my original workaround was still required.
There's something specific about using mpegts that leads to this
problem.

I'm not familiar with what parts of what metadata get respected in
different contextx For example, the out.ts, this command line produces,
is reported as having the right aspect ratio by mediainfo, but the
wrong one by ffprobe (and then plays back wrong, obviously).

Modifying the darWidth and darHeight leads to changes that are visible
in mediainfo, and are cumulative with respect to whatever bug is
hitting you here.

So, theory - there was a bug where nvenc was distorting the DAR and
that bug is now fixed. It seems like there is now some other bug, or
perhaps there always was a bug, which is modifying the SAR. Perhaps it
is doing it all the time but in other containers, container level
metadata is overriding it so it never becomes an issue.

Sounds like a great time for an nvidia dev to chime in :-)

--phil
Miroslav Slugeň Feb. 13, 2017, 4:33 p.m.
Dne 13.2.2017 v 17:07 Philip Langdale napsal(a):
> On Mon, 13 Feb 2017 07:21:51 -0800
> Philip Langdale <philipl@overt.org> wrote:
>
>> On Mon, 13 Feb 2017 08:52:34 +0100
>> Miroslav Slugeň <thunder.m@email.cz> wrote:
>>
>>>>     
>>> I am using current STABLE drivers 375.26, because BETA drivers
>>> 378.09 caused some crashes while encoding on NVENC.
>>>
>>> I tested this on BETA drivers too and it is still same.
>>>
>>> Original workaround is not working anymore :(
>>>
>>> INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main) ([2][0][0][0] /
>>> 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25
>>> fps, 25 tbr, 90k tbn, 50 tbc
>>>
>>> OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] /
>>> 0x001B), yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25
>>> fps, 25 tbr, 90k tbn, 50 tbc
>>>
>>> COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid -i
>>> "in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f mpegts
>>> "out.ts"
>>>
>>> Also someone else is complaining about this issue:
>>> http://superuser.com/questions/1174097/ffmpegnvenc-encoding-strange-aspect-ratio
>>>
>>> M.
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> Can you point me to a sample that you see this behaviour with? I
>> cannot reproduce with DVD sources here, which is where I saw the
>> original problem.
> You sent me a sample and I tried it out. I was able to reproduce your
> problem, but it is not the original problem, and I wonder what is
> really going on.
>
> If you take your sample and your command line and then output to a
> different container (i tried mkv and mp4), it's all correct - and it
> would not be correct if my original workaround was still required.
> There's something specific about using mpegts that leads to this
> problem.
>
> I'm not familiar with what parts of what metadata get respected in
> different contextx For example, the out.ts, this command line produces,
> is reported as having the right aspect ratio by mediainfo, but the
> wrong one by ffprobe (and then plays back wrong, obviously).
>
> Modifying the darWidth and darHeight leads to changes that are visible
> in mediainfo, and are cumulative with respect to whatever bug is
> hitting you here.
>
> So, theory - there was a bug where nvenc was distorting the DAR and
> that bug is now fixed. It seems like there is now some other bug, or
> perhaps there always was a bug, which is modifying the SAR. Perhaps it
> is doing it all the time but in other containers, container level
> metadata is overriding it so it never becomes an issue.
>
> Sounds like a great time for an nvidia dev to chime in :-)
>
> --phil
>
I am sure that i know what is going on, NVENC is inserting wrong SPS VUI 
aspect_ratio_idc to h264 packets when you encode at resolution 720x576 
and 720x480

AR 16:9 will insert aspect_ratio_idc=4 but it should be 
aspect_ratio_idc=255, sar_width=64, sar_height=45 for 720x576
AR 4:3 will insert aspect_ratio_idc=2 but it should be 
aspect_ratio_idc=255, sar_width=16, sar_height=15 for 720x576

MP4 and MKV containers contains correct AR metadata information which 
all players should accept, but TS container has nothing like that so it 
relies on what is inside h264 SPS.

Miroslav Slugeň
+420 724 825 885
Philip Langdale Feb. 13, 2017, 5:26 p.m.
On 2017-02-13 08:33, Miroslav Slugeň wrote:
> I am sure that i know what is going on, NVENC is inserting wrong SPS
> VUI aspect_ratio_idc to h264 packets when you encode at resolution
> 720x576 and 720x480
> 
> AR 16:9 will insert aspect_ratio_idc=4 but it should be
> aspect_ratio_idc=255, sar_width=64, sar_height=45 for 720x576
> AR 4:3 will insert aspect_ratio_idc=2 but it should be
> aspect_ratio_idc=255, sar_width=16, sar_height=15 for 720x576
> 
> MP4 and MKV containers contains correct AR metadata information which
> all players should accept, but TS container has nothing like that so
> it relies on what is inside h264 SPS.

Makes sense. So, I think this really needs fixing inside the driver.
Having to do the whole parse/rewrite thing is horrible - and that's
even after rewriting it not to re-implement the parser.

--phil
Yogender Gupta Feb. 14, 2017, 12:32 p.m.
Hi Miroslav , Philip,

We will look at this issue, and provide a fix for the same. Would be great to send me and Sumit (also copied here) the command lines that you tried and the observations to repro at our end.

Thanks,
Yogender

-----Original Message-----
From: Miroslav Slugeň [mailto:thunder.m@email.cz] 

Sent: Monday, February 13, 2017 10:04 PM
To: Philip Langdale
Cc: FFmpeg development discussions and patches; Sumit Agarwal; Yogender Gupta
Subject: Re: [FFmpeg-devel] [PATCH] (for discussion): nvenc: fix wrong aspect ratio for 720x576 and 720x480 resolution

Dne 13.2.2017 v 17:07 Philip Langdale napsal(a):
> On Mon, 13 Feb 2017 07:21:51 -0800

> Philip Langdale <philipl@overt.org> wrote:

>

>> On Mon, 13 Feb 2017 08:52:34 +0100

>> Miroslav Slugeň <thunder.m@email.cz> wrote:

>>

>>>>     

>>> I am using current STABLE drivers 375.26, because BETA drivers

>>> 378.09 caused some crashes while encoding on NVENC.

>>>

>>> I tested this on BETA drivers too and it is still same.

>>>

>>> Original workaround is not working anymore :(

>>>

>>> INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main) ([2][0][0][0] / 

>>> 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 25 

>>> fps, 25 tbr, 90k tbn, 50 tbc

>>>

>>> OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 

>>> 0x001B), yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25 

>>> fps, 25 tbr, 90k tbn, 50 tbc

>>>

>>> COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid -i 

>>> "in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f mpegts 

>>> "out.ts"

>>>

>>> Also someone else is complaining about this issue:

>>> http://superuser.com/questions/1174097/ffmpegnvenc-encoding-strange-

>>> aspect-ratio

>>>

>>> M.

>>> _______________________________________________

>>> ffmpeg-devel mailing list

>>> ffmpeg-devel@ffmpeg.org

>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

>> Can you point me to a sample that you see this behaviour with? I 

>> cannot reproduce with DVD sources here, which is where I saw the 

>> original problem.

> You sent me a sample and I tried it out. I was able to reproduce your 

> problem, but it is not the original problem, and I wonder what is 

> really going on.

>

> If you take your sample and your command line and then output to a 

> different container (i tried mkv and mp4), it's all correct - and it 

> would not be correct if my original workaround was still required.

> There's something specific about using mpegts that leads to this 

> problem.

>

> I'm not familiar with what parts of what metadata get respected in 

> different contextx For example, the out.ts, this command line 

> produces, is reported as having the right aspect ratio by mediainfo, 

> but the wrong one by ffprobe (and then plays back wrong, obviously).

>

> Modifying the darWidth and darHeight leads to changes that are visible 

> in mediainfo, and are cumulative with respect to whatever bug is 

> hitting you here.

>

> So, theory - there was a bug where nvenc was distorting the DAR and 

> that bug is now fixed. It seems like there is now some other bug, or 

> perhaps there always was a bug, which is modifying the SAR. Perhaps it 

> is doing it all the time but in other containers, container level 

> metadata is overriding it so it never becomes an issue.

>

> Sounds like a great time for an nvidia dev to chime in :-)

>

> --phil

>

I am sure that i know what is going on, NVENC is inserting wrong SPS VUI aspect_ratio_idc to h264 packets when you encode at resolution 720x576 and 720x480

AR 16:9 will insert aspect_ratio_idc=4 but it should be aspect_ratio_idc=255, sar_width=64, sar_height=45 for 720x576 AR 4:3 will insert aspect_ratio_idc=2 but it should be aspect_ratio_idc=255, sar_width=16, sar_height=15 for 720x576

MP4 and MKV containers contains correct AR metadata information which all players should accept, but TS container has nothing like that so it relies on what is inside h264 SPS.

Miroslav Slugeň
+420 724 825 885


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
Philip Langdale Feb. 15, 2017, 3:49 a.m.
On Tue, 14 Feb 2017 12:32:07 +0000
Yogender Gupta <ygupta@nvidia.com> wrote:

> Hi Miroslav , Philip,
> 
> We will look at this issue, and provide a fix for the same. Would be
> great to send me and Sumit (also copied here) the command lines that
> you tried and the observations to repro at our end.

Here's the sample from Miroslav - I've stuck it on dropbox.

https://www.dropbox.com/s/1bjyhdzcuv1y357/dvb_mpegts_720x576_sar_nvenc.ts?dl=0

Here is the ffprobe report and the command line:

> >>> INPUT: Stream #0:0[0x401]: Video: mpeg2video (Main)
> >>> ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x576 [SAR
> >>> 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
> >>>
> >>> OUTPUT: Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 
> >>> 0x001B), yuv420p(progressive), 720x576 [SAR 16:11 DAR 20:11], 25 
> >>> fps, 25 tbr, 90k tbn, 50 tbc
> >>>
> >>> COMMAND: ffmpeg -deint adaptive -hwaccel cuvid -c:v mpeg2_cuvid
> >>> -i "in.ts" -y -c:v h264_nvenc -c:a copy -b:v 1M -preset hq -f
> >>> mpegts "out.ts"

and for completeness, Miroslav's analysis:

> I am sure that i know what is going on, NVENC is inserting wrong SPS
> VUI aspect_ratio_idc to h264 packets when you encode at resolution
> 720x576 and 720x480
> 
> AR 16:9 will insert aspect_ratio_idc=4 but it should be
> aspect_ratio_idc=255, sar_width=64, sar_height=45 for 720x576 AR 4:3
> will insert aspect_ratio_idc=2 but it should be aspect_ratio_idc=255,
> sar_width=16, sar_height=15 for 720x576
> 
> MP4 and MKV containers contains correct AR metadata information which
> all players should accept, but TS container has nothing like that so
> it relies on what is inside h264 SPS.

--phil

Patch hide | download patch | download mbox

From 2300c38e381ee9bf4af96e6c0f6d05193738bcab Mon Sep 17 00:00:00 2001
From: Miroslav Slugen <thunder.m@email.cz>
Date: Sun, 12 Feb 2017 22:19:02 +0100
Subject: [PATCH 1/1] nvenc: fix wrong aspect ratio for 720x576 and 720x480
 resolutions

---
 libavcodec/nvenc.c    |   4 +
 libavcodec/nvenc_ar.c | 414 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 418 insertions(+)
 create mode 100644 libavcodec/nvenc_ar.c

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 7005465..87284b7 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -31,6 +31,8 @@ 
 #include "libavutil/pixdesc.h"
 #include "internal.h"
 
+#include "nvenc_ar.c"
+
 #define NVENC_CAP 0x30
 #define IS_CBR(rc) (rc == NV_ENC_PARAMS_RC_CBR ||               \
                     rc == NV_ENC_PARAMS_RC_2_PASS_QUALITY ||    \
@@ -1613,6 +1615,8 @@  FF_ENABLE_DEPRECATION_WARNINGS
     if (res < 0)
         goto error2;
 
+    nvenc_sar_fix(avctx, pkt);
+
     av_free(slice_offsets);
 
     return 0;
diff --git a/libavcodec/nvenc_ar.c b/libavcodec/nvenc_ar.c
new file mode 100644
index 0000000..a45c7fb
--- /dev/null
+++ b/libavcodec/nvenc_ar.c
@@ -0,0 +1,414 @@ 
+
+typedef struct nal_unit {
+    AVPacket *pkt;
+
+    int reset;
+
+    uint8_t *b;
+    int bs;
+
+    int nal_ref_idc;
+    int nal_unit_type;
+
+    int profile_idc;
+    int level_idc;
+    int seq_parameter_set_id;
+
+    int chroma_format_idc;
+    int residual_colour_transform_flag;
+    int bit_depth_luma_minus8;
+    int bit_depth_chroma_minus8;
+    int qpprime_y_zero_transform_bypass_flag;
+    int seq_scaling_matrix_present_flag;
+    int seq_scaling_list_present_flag[8];
+    int* scaling_list_4x4[6];
+    int UseDefaultScalingMatrix4x4Flag[6];
+    int* scaling_list_8x8[2];
+    int UseDefaultScalingMatrix8x8Flag[2];
+
+
+    int log2_max_frame_num_minus4;
+    int pic_order_cnt_type;
+    int log2_max_pic_order_cnt_lsb_minus4;
+    int delta_pic_order_always_zero_flag;
+    int offset_for_non_ref_pic;
+    int offset_for_top_to_bottom_field;
+    int num_ref_frames_in_pic_order_cnt_cycle;
+    int offset_for_ref_frame[256];
+    int num_ref_frames;
+    int gaps_in_frame_num_value_allowed_flag;
+    int pic_width_in_mbs_minus1;
+    int pic_height_in_map_units_minus1;
+    int frame_mbs_only_flag;
+    int mb_adaptive_frame_field_flag;
+    int direct_8x8_inference_flag;
+    int frame_cropping_flag;
+    int frame_crop_left_offset;
+    int frame_crop_right_offset;
+    int frame_crop_top_offset;
+    int frame_crop_bottom_offset;
+    int vui_parameters_present_flag;
+
+    struct {
+        int aspect_ratio_info_present_flag;
+        int aspect_ratio_idc;
+        int sar_width;
+        int sar_height;
+        int overscan_info_present_flag;
+        int overscan_appropriate_flag;
+        int video_signal_type_present_flag;
+        int video_format;
+        int video_full_range_flag;
+        int colour_description_present_flag;
+        int colour_primaries;
+        int transfer_characteristics;
+        int matrix_coefficients;
+        int chroma_loc_info_present_flag;
+        int chroma_sample_loc_type_top_field;
+        int chroma_sample_loc_type_bottom_field;
+        int timing_info_present_flag;
+        int num_units_in_tick;
+        int time_scale;
+        int fixed_frame_rate_flag;
+        int nal_hrd_parameters_present_flag;
+        int vcl_hrd_parameters_present_flag;
+        int low_delay_hrd_flag;
+        int pic_struct_present_flag;
+        int bitstream_restriction_flag;
+        int motion_vectors_over_pic_boundaries_flag;
+        int max_bytes_per_pic_denom;
+        int max_bits_per_mb_denom;
+        int log2_max_mv_length_horizontal;
+        int log2_max_mv_length_vertical;
+        int num_reorder_frames;
+        int max_dec_frame_buffering;
+    } vui;
+
+} nal_unit;
+
+
+static int nal_unit_find (uint8_t* b, int bs, int* start, int* end) {
+    int i;
+
+    *start = 0;
+    *end = 0;
+
+    for (i = 0; i < bs; i++) {
+        if ((i + 3 <= bs) && (b[i] == 0 && b[i + 1] == 0 && b[i + 2] == 0x01)) {
+            i+= 3;
+            goto found;
+        }
+
+        if ((i + 4 <= bs) && (b[i] == 0 && b[i + 1] == 0 && b[i + 2] == 0 && b[i + 3] == 0x01)) {
+            i+= 4;
+            goto found;
+        }
+    }
+
+    goto not_found;
+
+found:
+    *start = i;
+
+    for (; i < bs; i++) {
+        if (((i + 3 <= bs) && (b[i] == 0 && b[i + 1] == 0 && b[i + 2] == 0x01)) ||
+            ((i + 4 <= bs) && (b[i] == 0 && b[i + 1] == 0 && b[i + 2] == 0 && b[i + 3] == 0x01))) break;
+    }
+
+    *end = i;
+    return (*end - *start);
+
+not_found:
+    return 0;
+}
+
+static int nal_unit_get (nal_unit *p, int bs) {
+    int i, pos, r = 0;
+
+    for (i = 0; (i < bs && p->bs > 0); i++) {
+        pos = (p->bs - 1) % 8;
+
+        r = (r << 1) | ((p->b[0] >> pos) & 0x1);
+
+        if (!pos) p->b++;
+        p->bs--;
+    }
+
+    return r;
+}
+
+static uint32_t nal_unit_get_ue (nal_unit *p){
+    int32_t r = 0;
+    int i = 0;
+
+    for (i = 0; ((nal_unit_get(p, 1) == 0) && (i < 32) && (p->bs > 0)); i++);
+
+    r = nal_unit_get(p, i);
+    r+= (1 << i) - 1;
+    return r;
+}
+
+static int32_t nal_unit_get_se (nal_unit *p) {
+    int32_t r = nal_unit_get_ue(p);
+    if (r & 0x01) {
+        r = (r+1)/2;
+    } else {
+        r = -(r/2);
+    }
+    return r;
+}
+
+static int nal_unit_set (nal_unit *p, int bs, int v) {
+    int i, pos;
+
+    for (i = 0; (i < bs && p->bs > 0); i++) {
+        pos = (p->bs - 1) % 8;
+
+        p->b[0] = (p->b[0] & (0xFF - (1 << pos))) | (((v >> ((bs - 1) - i)) & 0x1) << pos);
+
+        if (!pos) p->b++;
+        p->bs--;
+    }
+
+    return 0;
+}
+
+static void nal_unit_read_scaling_list(nal_unit *p, int* scalingList, int sizeOfScalingList, int useDefaultScalingMatrixFlag ) {
+    int j, lastScale = 8, nextScale = 8, delta_scale;
+
+    if (!scalingList) return;
+
+    for (j = 0; j < sizeOfScalingList; j++) {
+        if (nextScale != 0) {
+            delta_scale = nal_unit_get_se(p);
+            nextScale = (lastScale + delta_scale + 256) % 256;
+            useDefaultScalingMatrixFlag = (j == 0 && nextScale == 0);
+        }
+        scalingList[j] = (nextScale == 0) ? lastScale : nextScale;
+        lastScale = scalingList[j];
+    }
+}
+
+static int nal_unit_ar_set (AVCodecContext *avctx, nal_unit *p, uint8_t *buf) {
+    int i, pos, ret = 0;
+
+    pos = p->b - p->pkt->data;
+
+    /* alokujeme další paměť */
+    if ((ret = av_grow_packet(p->pkt, 4)) < 0) return ret;
+
+    /* posuneme všechna data za pozicí */
+    memmove(p->pkt->data + pos + 4, p->pkt->data + pos, p->pkt->size - pos);
+    p->pkt->size+= 4;
+
+    /* posuneme se na novou pozici aspect_ratio_idc */
+    p->b = p->pkt->data + (pos - 1);
+    p->bs+= 8 + 32;
+
+    /* musíme nastavit novou pozici */
+    for (i = 0; i < 5; i++) {
+        nal_unit_set(p, 8, buf[i]);
+    }
+
+    p->reset = 1;
+
+    return ret;
+}
+
+static int nal_unit_read_vui (AVCodecContext *avctx, nal_unit *p) {
+    char buf[5];
+
+    p->vui.aspect_ratio_info_present_flag = nal_unit_get(p, 1);
+    if (p->vui.aspect_ratio_info_present_flag) {
+        p->vui.aspect_ratio_idc = nal_unit_get(p, 8);
+
+        switch (p->vui.aspect_ratio_idc) {
+        /* 4:3 */
+        case 2:
+            buf[0] = 0xFF;
+            buf[1] = 0;
+            buf[2] = 0x10;
+            buf[3] = 0;
+            buf[4] = 0x0F;
+            nal_unit_ar_set(avctx, p, buf);
+            break;
+        /* 16:9 */
+        case 4:
+            buf[0] = 0xFF;
+            buf[1] = 0;
+            buf[2] = 0x40;
+            buf[3] = 0;
+            buf[4] = 0x2D;
+            nal_unit_ar_set(avctx, p, buf);
+            break;
+        case 0xFF:
+            p->vui.sar_width = nal_unit_get(p, 16);
+            p->vui.sar_height = nal_unit_get(p, 16);
+            break;
+        }
+
+        //av_log(avctx, AV_LOG_ERROR, "NAL AR idc: %d, sarw: %d, sarh: %d\n", p->vui.aspect_ratio_idc, p->vui.sar_width, p->vui.sar_height);
+    }
+    /* ... */
+
+    return 0;
+}
+
+static int nal_unit_read_sps (AVCodecContext *avctx, nal_unit *p) {
+    int i;
+
+    /* 8bit - profile_idc */
+    p->profile_idc = nal_unit_get(p, 8);
+    /* 6bit - constraint_setX_flags */
+    nal_unit_get(p, 1);
+    nal_unit_get(p, 1);
+    nal_unit_get(p, 1);
+    nal_unit_get(p, 1);
+    nal_unit_get(p, 1);
+    nal_unit_get(p, 1);
+    /* 2bit - reserved_zero */
+    nal_unit_get(p, 2);
+    /* 8bit - level_idc */
+    p->level_idc = nal_unit_get(p, 8);
+    p->seq_parameter_set_id = nal_unit_get_ue(p);
+
+    //av_log(avctx, AV_LOG_ERROR, "NAL profile: %d, level: %d, set_id: %d\n", p->profile_idc, p->level_idc, p->seq_parameter_set_id);
+
+    if (p->profile_idc == 100 || p->profile_idc == 110 ||
+        p->profile_idc == 122 || p->profile_idc == 144) {
+        p->chroma_format_idc = nal_unit_get_ue(p);
+        if (p->chroma_format_idc == 3) {
+            p->residual_colour_transform_flag = nal_unit_get(p, 1);
+        }
+        p->bit_depth_luma_minus8 = nal_unit_get_ue(p);
+        p->bit_depth_chroma_minus8 = nal_unit_get_ue(p);
+        p->qpprime_y_zero_transform_bypass_flag = nal_unit_get(p, 1);
+        p->seq_scaling_matrix_present_flag = nal_unit_get(p, 1);
+        if (p->seq_scaling_matrix_present_flag) {
+            for (i = 0; i < 8; i++) {
+                p->seq_scaling_list_present_flag[i] = nal_unit_get(p, 1);
+                if (p->seq_scaling_list_present_flag[i]) {
+                    if (i < 6) {
+                        nal_unit_read_scaling_list(p, p->scaling_list_4x4[i], 16, p->UseDefaultScalingMatrix4x4Flag[i]);
+                    } else {
+                        nal_unit_read_scaling_list(p, p->scaling_list_8x8[i - 6], 64, p->UseDefaultScalingMatrix8x8Flag[i - 6]);
+                    }
+                }
+            }
+        }
+    }
+    p->log2_max_frame_num_minus4 = nal_unit_get_ue(p);
+    p->pic_order_cnt_type = nal_unit_get_ue(p);
+    if (p->pic_order_cnt_type == 0) {
+        p->log2_max_pic_order_cnt_lsb_minus4 = nal_unit_get_ue(p);
+    } else if (p->pic_order_cnt_type == 1) {
+        p->delta_pic_order_always_zero_flag = nal_unit_get(p, 1);
+        p->offset_for_non_ref_pic = nal_unit_get_se(p);
+        p->offset_for_top_to_bottom_field = nal_unit_get_se(p);
+        p->num_ref_frames_in_pic_order_cnt_cycle = nal_unit_get_ue(p);
+        for(i = 0; i < p->num_ref_frames_in_pic_order_cnt_cycle; i++) {
+            p->offset_for_ref_frame[i] = nal_unit_get_se(p);
+        }
+    }
+    p->num_ref_frames = nal_unit_get_ue(p);
+    p->gaps_in_frame_num_value_allowed_flag = nal_unit_get(p, 1);
+
+    /* kontrola na rozlišení nutná */
+    p->pic_width_in_mbs_minus1 = nal_unit_get_ue(p);
+    p->pic_height_in_map_units_minus1 = nal_unit_get_ue(p);
+    if ((p->pic_width_in_mbs_minus1 != 44) ||
+        ((p->pic_height_in_map_units_minus1 != 35) && (p->pic_height_in_map_units_minus1 != 29))) goto done;
+
+    p->frame_mbs_only_flag = nal_unit_get(p, 1);
+    if (!p->frame_mbs_only_flag) {
+        p->mb_adaptive_frame_field_flag = nal_unit_get(p, 1);
+    }
+    p->direct_8x8_inference_flag = nal_unit_get(p, 1);
+    p->frame_cropping_flag = nal_unit_get(p, 1);
+    if(p->frame_cropping_flag) {
+        p->frame_crop_left_offset = nal_unit_get_ue(p);
+        p->frame_crop_right_offset = nal_unit_get_ue(p);
+        p->frame_crop_top_offset = nal_unit_get_ue(p);
+        p->frame_crop_bottom_offset = nal_unit_get_ue(p);
+    }
+    p->vui_parameters_present_flag = nal_unit_get(p, 1);
+    if (p->vui_parameters_present_flag) {
+        nal_unit_read_vui(avctx, p);
+    }
+    //read_rbsp_trailing_bits
+
+done:
+    return 0;
+}
+
+static int nal_unit_read (AVCodecContext *avctx, nal_unit *p) {
+
+    /* 1bit - forbidden_zero_bit */
+    if (nal_unit_get(p, 1)) goto done;
+
+    /* 2bit - nal_ref_idc */
+    p->nal_ref_idc = nal_unit_get(p, 2);
+
+    /* 5bit - nal_unit_type */
+    p->nal_unit_type = nal_unit_get(p, 5);
+
+    switch (p->nal_unit_type) {
+    case 7:
+        //av_log(avctx, AV_LOG_ERROR, "NAL SPS\n");
+        nal_unit_read_sps(avctx, p);
+        break;
+    default:
+        goto done;
+    }
+
+done:
+    return 0;
+}
+
+static int nvenc_sar_fix (AVCodecContext *avctx, AVPacket *pkt) {
+    int i, ret, size, len, start, stop;
+    uint8_t *data = NULL;
+    nal_unit nal;
+
+reset:
+    i = 0;
+    data = pkt->data;
+    size = pkt->size;
+
+next:
+    if (size <= 0) goto done;
+
+    if ((ret = nal_unit_find(data, size, &start, &stop)) < 0) return ret;
+
+    len = ret;
+    if (len + start <= 0) goto done;
+
+    //av_log(avctx,AV_LOG_ERROR,"found %d NAL uint: %d - %d, length: %d\n", i, start, stop, len);
+
+    data+= start;
+    size-= start;
+
+    /* prázdná NAL */
+    if (len <= 0) {
+        i++;
+        goto next;
+    }
+
+    memset(&nal, 0, sizeof(nal_unit));
+
+    nal.pkt = pkt;
+    nal.b = data;
+    nal.bs = len << 3;
+
+    if ((ret = nal_unit_read(avctx, &nal)) < 0) return ret;
+
+    if (nal.reset) goto reset;
+
+    data+= len;
+    size-= len;
+    i++;
+    goto next;
+
+done:
+    return 0;
+}
-- 
2.1.4