diff options
| author | Jeremy Dorfman <jdorfman@google.com> | 2022-11-15 14:30:23 -0500 |
|---|---|---|
| committer | Michael Niedermayer <michael@niedermayer.cc> | 2022-11-20 21:23:57 +0100 |
| commit | ce566281f943227480b5441401e90ee8a843730f (patch) | |
| tree | 9042f40b10549660100a00ac8ce1e9545b98aed1 /libswscale | |
| parent | 0871cb9499242cb323c245599f3e7e763806e85b (diff) | |
| download | ffmpeg-ce566281f943227480b5441401e90ee8a843730f.tar.gz | |
swscale/input: Use unsigned intermediates in rgb64ToUV_c_template
Large rgb2yuv tables and high pixel values cause the intermediate
int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
behavior. This causes libswscale built with LLVM -fsanitize=undefined to
assert. Using unsigned integers instead has defined behavior and
produces identical results, and makes rgb64ToUV_c_template match
rgb64ToY_c_template.
Fixes: signed integer overflow
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Diffstat (limited to 'libswscale')
| -rw-r--r-- | libswscale/input.c | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; |