diff mbox

[FFmpeg-devel,3/3] tiff: fix overflows when calling av_readuce

Message ID a4805212-250e-740f-9520-fab5e006b452@googlemail.com
State Superseded
Headers show

Commit Message

Andreas Cadhalpun Dec. 12, 2016, 11:50 p.m. UTC
The arguments of av_reduce are signed, so the cast to uint64_t is misleading.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
---
 libavcodec/tiff.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

Comments

Michael Niedermayer Dec. 13, 2016, 12:32 a.m. UTC | #1
On Tue, Dec 13, 2016 at 12:50:19AM +0100, Andreas Cadhalpun wrote:
> The arguments of av_reduce are signed, so the cast to uint64_t is misleading.
> 
> Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
> ---
>  libavcodec/tiff.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/tiff.c b/libavcodec/tiff.c
> index 4721e94..12ef419 100644
> --- a/libavcodec/tiff.c
> +++ b/libavcodec/tiff.c
> @@ -772,9 +772,16 @@ static void set_sar(TiffContext *s, unsigned tag, unsigned num, unsigned den)
>      int offset = tag == TIFF_YRES ? 2 : 0;
>      s->res[offset++] = num;
>      s->res[offset]   = den;
> -    if (s->res[0] && s->res[1] && s->res[2] && s->res[3])
> +    if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) {
> +        uint64_t num = s->res[2] * (uint64_t)s->res[1];
> +        uint64_t den = s->res[0] * (uint64_t)s->res[3];
> +        if (num > INT64_MAX || den > INT64_MAX) {
> +            num = num >> 1;
> +            den = den >> 1;
> +        }

this can make one of them 0, in fact i think even if they arent 0
the sample_aspect_ratio can be  after reduce
should they be checked after all that instead of before ?


[...]
diff mbox

Patch

diff --git a/libavcodec/tiff.c b/libavcodec/tiff.c
index 4721e94..12ef419 100644
--- a/libavcodec/tiff.c
+++ b/libavcodec/tiff.c
@@ -772,9 +772,16 @@  static void set_sar(TiffContext *s, unsigned tag, unsigned num, unsigned den)
     int offset = tag == TIFF_YRES ? 2 : 0;
     s->res[offset++] = num;
     s->res[offset]   = den;
-    if (s->res[0] && s->res[1] && s->res[2] && s->res[3])
+    if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) {
+        uint64_t num = s->res[2] * (uint64_t)s->res[1];
+        uint64_t den = s->res[0] * (uint64_t)s->res[3];
+        if (num > INT64_MAX || den > INT64_MAX) {
+            num = num >> 1;
+            den = den >> 1;
+        }
         av_reduce(&s->avctx->sample_aspect_ratio.num, &s->avctx->sample_aspect_ratio.den,
-                  s->res[2] * (uint64_t)s->res[1], s->res[0] * (uint64_t)s->res[3], INT32_MAX);
+                  num, den, INT32_MAX);
+    }
 }
 
 static int tiff_decode_tag(TiffContext *s, AVFrame *frame)