py/mpz: Avoid undefined behavior at integer overflow in mpz_hash.

Before this, ubsan would detect a problem when executing
hash(006699999999999999999999999999999999999999999999999999999999999999999999)

    ../../py/mpz.c:1539:20: runtime error: left shift of 1067371580458 by
    32 places cannot be represented in type 'mp_int_t' (aka 'long')

When the overflow does occur it now happens as defined by the rules of
unsigned arithmetic.
crypto-aes
Jeff Epler 5 years ago committed by Damien George
parent 60eb5305f6
commit c4dafcef4f

@ -1532,7 +1532,7 @@ mpz_t *mpz_mod(const mpz_t *lhs, const mpz_t *rhs) {
// must return actual int value if it fits in mp_int_t
mp_int_t mpz_hash(const mpz_t *z) {
mp_int_t val = 0;
mp_uint_t val = 0;
mpz_dig_t *d = z->dig + z->len;
while (d-- > z->dig) {

Loading…
Cancel
Save