Python does not provide PyLong_FromIntMax_t() or PyLong_FromUintMax_t() function

[ssbr]

BPO	17870
Nosy	@mdickinson, @pitrou, @vstinner, @benjaminp, @ssbr, @serhiy-storchaka, @1st1, @MojoVampire
PRs	[WIP] bpo-17870: Add support for C intmax_t type #857 convert from long long to PyLong losslessly #1106
Files	intmax_t.patch use_intmax_t.patch intmax_t-2.patch intmax_t-3.patch intmax_t-4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-04-29.20:57:04.995>
labels = ['type-feature']
title = 'Python does not provide PyLong_FromIntMax_t() or PyLong_FromUintMax_t() function'
updated_at = <Date 2017-06-15.14:08:50.685>
user = 'https://github.com/ssbr'

bugs.python.org fields:

activity = <Date 2017-06-15.14:08:50.685>
actor = 'Devin Jeanpierre'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2013-04-29.20:57:04.995>
creator = 'Devin Jeanpierre'
dependencies = []
files = ['30082', '30085', '30240', '30745', '45641']
hgrepos = []
issue_num = 17870
keywords = ['patch']
message_count = 39.0
messages = ['188103', '188105', '188106', '188111', '188112', '188114', '188118', '188126', '188142', '188149', '188168', '188189', '188194', '188196', '188689', '188704', '189051', '189066', '189418', '190693', '190696', '192163', '192182', '229533', '229535', '281706', '281707', '281708', '290684', '296040', '296042', '296043', '296054', '296066', '296089', '296093', '296094', '296095', '296096']
nosy_count = 8.0
nosy_names = ['mark.dickinson', 'pitrou', 'vstinner', 'benjamin.peterson', 'Devin Jeanpierre', 'serhiy.storchaka', 'yselivanov', 'josh.r']
pr_nums = ['857', '1106']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17870'
versions = ['Python 3.5']

[ssbr]

As far as I can tell, the only safe and correct way to convert a (for example) intptr_t to a python long is something akin to the following:

 size_t repsize = sizeof(intmax_t)*8 + 2;
 char i_buf[repsize]; // enough to fit base 2 with sign, let alone base 1
 snprintf(i_buf, repsize, "%ij", (intmax_t) myinteger);
 return PyLong_FromString(i_buf, NULL, 10);

Does this not seem absurd?

PyLong_FromIntMax_t(myinteger) would be great. Or maybe even better would be PyLong_FromBytes(&myinteger, sizeof(myinteger)) ?

This is problematic because many things that can interact with the Python C-API do not use the C types like "long long" or "int". Instead they might use the equivalents of intptr_t and int32_t, which are more reliably sized.

[vstinner]

Did you try PyLong_FromVoidPtr()?

[ssbr]

PyLong_FromVoidPtr works for uintptr_t, but not intptr_t.

[vstinner]

"PyLong_FromVoidPtr works for uintptr_t, but not intptr_t."

Ok correct. You should use something like:

PyObject*
PyLong_FromIntptr_t(intptr_t value)
{
 if (sizeof(intptr_t) == sizeof(long))
 return PyLong_FromLong(value);
 else{
 assert(sizeof(intptr_t) <= sizeof(PY_LONG_LONG));
 return PyLong_FromLongLong((PY_LONG_LONG)value);
 }
}

The "if (sizeof(intptr_t) == sizeof(long))" should be optimized by your compiler.

I don't know if Python should provide such function. What is your use case for intptr_t?

It looks like intptr_t is only really used in the implementation of os.spawnv() and os.spawnve(). Extract:

#if SIZEOF_LONG == SIZEOF_VOID_P
 return Py_BuildValue("l", (long) spawnval);
#else
 return Py_BuildValue("L", (PY_LONG_LONG) spawnval);
#endif

[ssbr]

Isn't it possible for a >64-bit architecture to have intptr_t be wider than long long?

As for my use-case, I am wrapping the C-API for Rust. Rust can call and be called by C (and therefore Python), but a Rust "int" is a C "intptr_t", and a Rust "uint" is a C "uintptr_t".

[ssbr]

With regards to the title change, I would prefer a FromIntMax_t (and FromUintMax_t) to a FromIntPtr_t. The former covers every use case of the latter, and more.

[vstinner]

The problem is to support platforms not providing a intmax_t type.

I don't know if the following C code would be a good "approximation" of the maximum integral value (signed/unsigned).

#ifdef HAVE_INTMAX_T
typedef intmax_t Py_intmax_t;
typedef uintmax_t Py_uintmax_t;
 
#elif defined(HAVE_LONG_LONG) && (SIZEOF_VOID_P <= SIZEOF_LONG_LONG)
typedef PY_LONG_LONG Py_intmax_t;
typedef unsigned PY_LONG_LONG Py_uintmax_t;

#elif SIZEOF_VOID_P <= SIZEOF_LONG
typedef long Py_intmax_t;
typedef unsigned long Py_uintmax_t;

#elif SIZEOF_VOID_P <= SIZEOF_INT
typedef int Py_intmax_t;
typedef unsigned int Py_uintmax_t;

#else
# error "Python needs a typedef for Py_intmax_t in pyport.h."
#endif /* HAVE_INTMAX_T */

If it is not, conversion from/to other types like off_t, time_t, pid_t or uid_t would loose information.

It is the same question than yours: is there a platform with an integer type wider than a pointer (intptr_t/void*)?

[serhiy-storchaka]

It is the same question than yours: is there a platform with an integer type wider than a pointer (intptr_t/void*)?

It's x86. sizeof(void*) == 4, sizeof(long long) == 8.

[vstinner]

> It is the same question than yours: is there a platform with an integer type wider than a pointer (intptr_t/void*)?

It's x86. sizeof(void*) == 4, sizeof(long long) == 8.

Ah yes. So "SIZEOF_VOID_P <=" is not a good test. File position
(off_t) size can be 64 bit on a 32-bit system (like Linux/i386 and
Win32), whereas pointers size is 32 bit.

I don't think that we can rely on the availability of PY_LONG_LONG, it
may depend on the compiler or even on the compiler options.

IMO, if we decide to add functions for intmax_t and uintmax_t types,
the safest option is to only define functions if the real C type
(intmax_t and uintmax_t types) is available. I mean, we should not
guess intmax_t ourself, or we may choose the wrong type (too small
type).

[vstinner]

Or maybe even better would be PyLong_FromBytes(&myinteger, sizeof(myinteger)) ?

There is a private function:

obj = _PyLong_FromByteArray((const unsigned char*)&myinteger, sizeof(myinteger), PY_LITTLE_ENDIAN, is_signed);

Where PY_LITTLE_ENDIAN is 1 on little-endian platform, and is_signed should be 0 or 1.

This function is inefficient (slow). It would be better to have a functions for intmax_t and uintmax_t types. Such functions would be enough to support other "recent" C types: intptr_t, uintptr_t, ptrdiff_t, etc. We may also add support for these types in PyArg_Parse* (ex: with "j" and "J" types, intmax_t and uintmax_t).

[vstinner]

Some Google searchs told me that no CPU support 128-bit integer and
using 64-bit integer as intmax_t is safe.

GCC has a __int128 type, but I didn't find on which platform it is
supported, nor if intmax_t is __int128 in this case.

Microsoft Visual Studio has stdint.h since its version 2010 (which is
the version required to compile Python 3.4 on Windows according to the
Python developer guide).

I propose a safer definition of Py_intmax_t:

#ifdef HAVE_UINTMAX_T
typedef uintmax_t Py_uintmax_t;
typedef intmax_t Py_intmax_t;

#elif SIZEOF_SIZE_T >= 8
typedef size_t Py_uintmax_t;
typedef Py_ssize_t Py_intmax_t;

#elif defined(HAVE_LONG_LONG) && SIZEOF_LONG_LONG >= 8
typedef unsigned PY_LONG_LONG Py_uintmax_t;
typedef PY_LONG_LONG Py_intmax_t;

#else
# error "Python needs a typedef for Py_uintmax_t in pyport.h."
#endif

I don't think that a fallback on the long type is required, testing
size_t should be enough.

At least, the compilation fails if the Py_intmax_t type cannot be defined.

Having generic PyLong_FromUintMax_t() and PyLong_AsUintMax_t()
functions (and signed version) would simplify the support of other OS
types with variable size: time_t, clock_t, pid_t, gid_t, uid_t, off_t,
etc.

[vstinner]

Here is a first patch adding the following functions:

 PyObject* PyLong_FromIntMax_t(intmax_t v);
 PyObject* PyLong_FromUintMax_t(uintmax_t v);
 intmax_t PyLong_AsIntMax_t(PyObject *pylong);
 uintmax_t PyLong_AsUintMax_t(PyObject *pylong);

I used AC_TYPE_INTMAX_T and AC_TYPE_UINTMAX_T in configure.ac, so intmax_t and uintmax_t are defined if the platform does not provide them. (I guess that these types are defined in pyconfig.h.) On Windows, Visual Studio 2010 supports stdint.h, and so these types are available.

autoheader && autoconf must be called to regenerate configure script and pyconfig.h.in template.