Raise ValueError rather of OverflowError in PyLong_AsUnsignedLong()

[serhiy-storchaka]

BPO	29834
Nosy	@rhettinger, @serhiy-storchaka, @orenmn

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-17.08:44:59.699>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Raise ValueError rather of OverflowError in PyLong_AsUnsignedLong()'
updated_at = <Date 2017-03-18.09:01:27.125>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-03-18.09:01:27.125>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2017-03-17.08:44:59.699>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 29834
keywords = []
message_count = 5.0
messages = ['289748', '289752', '289796', '289797', '289798']
nosy_count = 3.0
nosy_names = ['rhettinger', 'serhiy.storchaka', 'Oren Milman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29834'
versions = ['Python 3.7']

Linked PRs

• gh-74020: Raise ValueError for negative values converted to unsigned #121114

[serhiy-storchaka]

OverflowError is raised when Python integer doesn't fit in C integer type due to platform limitations. Different platforms have different limits. But in PyLong_AsUnsignedLong() only the upper limit is platform-depended. Negative integers always are not accepted. PyLong_AsUnsignedLong() is used for values that can be only non-negative. I think that ValueError is more appropriate in this case than OverflowError.

[orenmn]

note that there are functions that rely on the fact that
PyLong_AsUnsignedLong currently raises OverflowError for both cases.

such functions would probably use
PyErr_ExceptionMatches(PyExc_OverflowError)
or something like
PyErr_GivenExceptionMatches(err, PyExc_OverflowError)

_Py_Uid_Converter() (in Modules/posixmodule.c) is an example, but
ISTM there aren't many such functions.

however, this is only in cpython's C code. I don't know how many
functions in cpython's python code rely on this.

also, maybe there is a lot of user code that rely on this?

[rhettinger]

Strong -1 on this. For zero benefit, this breaks everything (within Python and third-party code) that ever relied on the documented behavior , https://docs.python.org/3/c-api/long.html#c.PyLong_AsUnsignedLong .

The cause perfectly fits the definition of an OverflowError:

class OverflowError(ArithmeticError)
 | Result too large to be represented.

The time to challenge API design decisions is when they are created, not after they've been published and relied upon for over decade.

Also, we really don't everyone who writes cross-platform code and tests to have to catch both exceptions because they don't know which version is being run. This creates yet another barrier to upgrading Python and as far as I can tell isn't solving any reported user problem. Instead, it is second-guessing design decisions made long ago.

[rhettinger]

Looking back in time, this API isn't as old as I thought. The other concerns about breaking a published API still stand.

[serhiy-storchaka]

_Py_Uid_Converter() in Modules/posixmodule.c is not an example because it calls PyLong_AsUnsignedLong() only for positive integers > LONG_MAX.

PyLong_AsUnsignedLong() is used not much. The motivation of this issue was that if use PyLong_AsUnsignedLong() for converting non-negative by its nature values ValueError can be more appropriate since this limitation is not platform or implementation dependent. Strictly speaking raising OverflowError for negative values doesn't fits the definition of an OverflowError, since the result is not large at all.

I was going to investigate all usages of PyLong_AsUnsignedLong() and if in majority of them ValueError is appropriate and desirable, changing the exception type at that level can make the implementation simpler.

[serhiy-storchaka]

Argument Clinic converters unsigned_char(bitwise=False), unsigned_short(bitwise=False), unsigned_int(bitwise=False), unsigned_long(bitwise=False), unsigned_long_long(bitwise=False), and size_t raise ValueError for negative integer. It is impossible to implement them using the limited C API.

Many code converts Python integer to signed C type and then raise ValueError if it is negative (see for example handling maxlen in deque()). So, negative value can cause either OverflowError or ValueError, depending on platform.