bug #8663 cant use bigint unsigned as input to cast

in the case of the overflow in the decimal->integer conversion
we didn't return the proper boundary value, but just the result
of the conversion we calculated on the moment of the error


mysql-test/r/bigint.result:
  bug #8663 cant use bigint unsigned as input to cast
  test result fixed
mysql-test/t/bigint.test:
  bug #8663 cant use bigint unsigned as input to cast
  test case
strings/decimal.c:
  bug #8663 cant use bigint unsigned as input to cast
  decimal->int conversion fixed to return proper boundary value
  in the case of overflow

1 files changed, 5 insertions, 1 deletions

diff --git a/strings/decimal.c b/strings/decimal.c
index 65db68b1b59..1ae75167794 100644
--- a/strings/decimal.c
+++ b/strings/decimal.c
@@ -1083,7 +1083,11 @@ int decimal2longlong(decimal_t *from, longlong *to)
     x=x*DIG_BASE - *buf++;
     if (unlikely(y < (LONGLONG_MIN/DIG_BASE) || x > y))
     {
-      *to= from->sign ? y : -y;
+      /*
+        the decimal is bigger than any possible integer
+        return border integer depending on the sign
+      */
+      *to= from->sign ? LONGLONG_MIN : LONGLONG_MAX;
       return E_DEC_OVERFLOW;
     }
   }