| Message ID | 20180326055118.32367-1-Pengfei.Qu@intel.com |
|---|---|
| State | New |
| Headers | show |
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces@ffmpeg.org] On Behalf > Of Pengfei Qu > Sent: Monday, March 26, 2018 1:51 PM > To: ffmpeg-devel@ffmpeg.org > Cc: Qu, Pengfei <pengfei.qu@intel.com> > Subject: [FFmpeg-devel] [V2 1/2] lavc/vaapi_encode: fix the caculation > overflow > > this fix the overflow during the caculation before value assignment. > > Signed-off-by: Pengfei Qu <Pengfei.Qu@intel.com> > --- > libavcodec/vaapi_encode.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c index > 36c85a3..78347d4 100644 > --- a/libavcodec/vaapi_encode.c > +++ b/libavcodec/vaapi_encode.c > @@ -1168,9 +1168,9 @@ static av_cold int > vaapi_encode_init_rate_control(AVCodecContext *avctx) > rc_target_percentage = 100; > } else { > rc_bits_per_second = avctx->rc_max_rate; > - rc_target_percentage = (avctx->bit_rate * 100) / > rc_bits_per_second; > + rc_target_percentage = (unsigned long)(avctx->bit_rate * > + 100) / rc_bits_per_second; > } > - rc_window_size = (hrd_buffer_size * 1000) / avctx->bit_rate; > + rc_window_size = (unsigned long)(hrd_buffer_size * 1000) / > + avctx->bit_rate; Should better to use " uint64_t". > } > > ctx->rc_params.misc.type = VAEncMiscParameterTypeRateControl; > -- > 2.9.3
diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c index 36c85a3..78347d4 100644 --- a/libavcodec/vaapi_encode.c +++ b/libavcodec/vaapi_encode.c @@ -1168,9 +1168,9 @@ static av_cold int vaapi_encode_init_rate_control(AVCodecContext *avctx) rc_target_percentage = 100; } else { rc_bits_per_second = avctx->rc_max_rate; - rc_target_percentage = (avctx->bit_rate * 100) / rc_bits_per_second; + rc_target_percentage = (unsigned long)(avctx->bit_rate * 100) / rc_bits_per_second; } - rc_window_size = (hrd_buffer_size * 1000) / avctx->bit_rate; + rc_window_size = (unsigned long)(hrd_buffer_size * 1000) / avctx->bit_rate; } ctx->rc_params.misc.type = VAEncMiscParameterTypeRateControl;
this fix the overflow during the caculation before value assignment. Signed-off-by: Pengfei Qu <Pengfei.Qu@intel.com> --- libavcodec/vaapi_encode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)