| Message ID | a4805212-250e-740f-9520-fab5e006b452@googlemail.com |
|---|---|
| State | Superseded |
| Headers | show |
On Tue, Dec 13, 2016 at 12:50:19AM +0100, Andreas Cadhalpun wrote: > The arguments of av_reduce are signed, so the cast to uint64_t is misleading. > > Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com> > --- > libavcodec/tiff.c | 11 +++++++++-- > 1 file changed, 9 insertions(+), 2 deletions(-) > > diff --git a/libavcodec/tiff.c b/libavcodec/tiff.c > index 4721e94..12ef419 100644 > --- a/libavcodec/tiff.c > +++ b/libavcodec/tiff.c > @@ -772,9 +772,16 @@ static void set_sar(TiffContext *s, unsigned tag, unsigned num, unsigned den) > int offset = tag == TIFF_YRES ? 2 : 0; > s->res[offset++] = num; > s->res[offset] = den; > - if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) > + if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) { > + uint64_t num = s->res[2] * (uint64_t)s->res[1]; > + uint64_t den = s->res[0] * (uint64_t)s->res[3]; > + if (num > INT64_MAX || den > INT64_MAX) { > + num = num >> 1; > + den = den >> 1; > + } this can make one of them 0, in fact i think even if they arent 0 the sample_aspect_ratio can be after reduce should they be checked after all that instead of before ? [...]
diff --git a/libavcodec/tiff.c b/libavcodec/tiff.c index 4721e94..12ef419 100644 --- a/libavcodec/tiff.c +++ b/libavcodec/tiff.c @@ -772,9 +772,16 @@ static void set_sar(TiffContext *s, unsigned tag, unsigned num, unsigned den) int offset = tag == TIFF_YRES ? 2 : 0; s->res[offset++] = num; s->res[offset] = den; - if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) + if (s->res[0] && s->res[1] && s->res[2] && s->res[3]) { + uint64_t num = s->res[2] * (uint64_t)s->res[1]; + uint64_t den = s->res[0] * (uint64_t)s->res[3]; + if (num > INT64_MAX || den > INT64_MAX) { + num = num >> 1; + den = den >> 1; + } av_reduce(&s->avctx->sample_aspect_ratio.num, &s->avctx->sample_aspect_ratio.den, - s->res[2] * (uint64_t)s->res[1], s->res[0] * (uint64_t)s->res[3], INT32_MAX); + num, den, INT32_MAX); + } } static int tiff_decode_tag(TiffContext *s, AVFrame *frame)
The arguments of av_reduce are signed, so the cast to uint64_t is misleading. Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com> --- libavcodec/tiff.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)