diff mbox series

[FFmpeg-devel,v5,1/9] cbs_av1: fix incorrect data type

Message ID 20211012082404.31639-1-fei.w.wang@intel.com
State New
Headers show
Series [FFmpeg-devel,v5,1/9] cbs_av1: fix incorrect data type
Related show

Checks

Context Check Description
andriy/make_x86 success Make finished
andriy/make_fate_x86 success Make fate finished
andriy/make_ppc success Make finished
andriy/make_fate_ppc success Make fate finished

Commit Message

Wang, Fei W Oct. 12, 2021, 8:23 a.m. UTC
Since order_hint_bits_minus_1 range is 0~7, cur_frame_hint can be
most 128. And similar return value for cbs_av1_get_relative_dist.
So if plus them and use int8_t for the result may lose its precision.

Signed-off-by: Fei Wang <fei.w.wang@intel.com>
---
update:
1. move additional film grain frame from av1dec.c to vaapi_av1.c
2. patch 3~5 can fix clip:
https://drive.google.com/file/d/1Qdx_18_BFcFf_5_XXSZOVohLpaAb-yIw/view?usp=sharing

 libavcodec/cbs_av1_syntax_template.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Wang, Fei W Oct. 15, 2021, 2:04 a.m. UTC | #1
On Tue, 2021-10-12 at 16:23 +0800, Fei Wang wrote:
> Since order_hint_bits_minus_1 range is 0~7, cur_frame_hint can be
> most 128. And similar return value for cbs_av1_get_relative_dist.
> So if plus them and use int8_t for the result may lose its precision.
> 
> Signed-off-by: Fei Wang <fei.w.wang@intel.com>
> ---
> update:
> 1. move additional film grain frame from av1dec.c to vaapi_av1.c
> 2. patch 3~5 can fix clip:
> https://drive.google.com/file/d/1Qdx_18_BFcFf_5_XXSZOVohLpaAb-yIw/view?usp=sharing
> 
>  libavcodec/cbs_av1_syntax_template.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavcodec/cbs_av1_syntax_template.c
> b/libavcodec/cbs_av1_syntax_template.c
> index 6fe6e9a4f3..d98d3d42de 100644
> --- a/libavcodec/cbs_av1_syntax_template.c
> +++ b/libavcodec/cbs_av1_syntax_template.c
> @@ -355,7 +355,7 @@ static int
> FUNC(set_frame_refs)(CodedBitstreamContext *ctx, RWContext *rw,
>          AV1_REF_FRAME_ALTREF2, AV1_REF_FRAME_ALTREF
>      };
>      int8_t ref_frame_idx[AV1_REFS_PER_FRAME],
> used_frame[AV1_NUM_REF_FRAMES];
> -    int8_t shifted_order_hints[AV1_NUM_REF_FRAMES];
> +    int16_t shifted_order_hints[AV1_NUM_REF_FRAMES];
>      int cur_frame_hint, latest_order_hint, earliest_order_hint, ref;
>      int i, j;
>  

Hi James,
Could you help to review this version? This fixed your comments in V4.
And we tested the 3th patch with DXVA, which can also fix the clip I
shared.

Fei
Thanks
James Almer Oct. 16, 2021, 10:03 p.m. UTC | #2
On 10/14/2021 11:04 PM, Wang, Fei W wrote:
> On Tue, 2021-10-12 at 16:23 +0800, Fei Wang wrote:
>> Since order_hint_bits_minus_1 range is 0~7, cur_frame_hint can be
>> most 128. And similar return value for cbs_av1_get_relative_dist.
>> So if plus them and use int8_t for the result may lose its precision.
>>
>> Signed-off-by: Fei Wang <fei.w.wang@intel.com>
>> ---
>> update:
>> 1. move additional film grain frame from av1dec.c to vaapi_av1.c
>> 2. patch 3~5 can fix clip:
>> https://drive.google.com/file/d/1Qdx_18_BFcFf_5_XXSZOVohLpaAb-yIw/view?usp=sharing
>>
>>   libavcodec/cbs_av1_syntax_template.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/libavcodec/cbs_av1_syntax_template.c
>> b/libavcodec/cbs_av1_syntax_template.c
>> index 6fe6e9a4f3..d98d3d42de 100644
>> --- a/libavcodec/cbs_av1_syntax_template.c
>> +++ b/libavcodec/cbs_av1_syntax_template.c
>> @@ -355,7 +355,7 @@ static int
>> FUNC(set_frame_refs)(CodedBitstreamContext *ctx, RWContext *rw,
>>           AV1_REF_FRAME_ALTREF2, AV1_REF_FRAME_ALTREF
>>       };
>>       int8_t ref_frame_idx[AV1_REFS_PER_FRAME],
>> used_frame[AV1_NUM_REF_FRAMES];
>> -    int8_t shifted_order_hints[AV1_NUM_REF_FRAMES];
>> +    int16_t shifted_order_hints[AV1_NUM_REF_FRAMES];
>>       int cur_frame_hint, latest_order_hint, earliest_order_hint, ref;
>>       int i, j;
>>   
> 
> Hi James,
> Could you help to review this version? This fixed your comments in V4.
> And we tested the 3th patch with DXVA, which can also fix the clip I
> shared.
> 
> Fei
> Thanks

Set applied.
diff mbox series

Patch

diff --git a/libavcodec/cbs_av1_syntax_template.c b/libavcodec/cbs_av1_syntax_template.c
index 6fe6e9a4f3..d98d3d42de 100644
--- a/libavcodec/cbs_av1_syntax_template.c
+++ b/libavcodec/cbs_av1_syntax_template.c
@@ -355,7 +355,7 @@  static int FUNC(set_frame_refs)(CodedBitstreamContext *ctx, RWContext *rw,
         AV1_REF_FRAME_ALTREF2, AV1_REF_FRAME_ALTREF
     };
     int8_t ref_frame_idx[AV1_REFS_PER_FRAME], used_frame[AV1_NUM_REF_FRAMES];
-    int8_t shifted_order_hints[AV1_NUM_REF_FRAMES];
+    int16_t shifted_order_hints[AV1_NUM_REF_FRAMES];
     int cur_frame_hint, latest_order_hint, earliest_order_hint, ref;
     int i, j;