ctypes: bit field data does not survive round trip

[matthiasgoergens]

Bit-fields in structures don't seem to give you back the data you put in?

from ctypes import Structure, c_uint, c_ulonglong, c_ushort


class Foo(Structure):
 _fields_ = [("A", c_uint, 1), ("B", c_ushort, 16)]


class Bar(Structure):
 _fields_ = [("A", c_ulonglong, 1), ("B", c_uint, 32)]


if __name__ == "__main__":
 for a in [Foo(), Bar()]:
 a.A = 0
 a.B = 1
 print(a.A, a.B)

The above should print

0 1
0 1

But it actually prints

$ python3.10 mini.py 
0 0
0 0

For comparison and to test my understanding, I expect the following C code to be equivalent to the Python code above:

#include<stdio.h>

struct Foo{
 unsigned int A: 1;
 unsigned short B: 16;
};

struct Bar{
 unsigned long long int A: 1;
 unsigned int B: 32;
};

int main(int argc, char** argv){
 struct Foo foo;
 foo.A = 0;
 foo.B = 1;
 printf("%d %d\n", foo.A, foo.B);

 struct Bar bar;
 bar.A = 0;
 bar.B = 1;
 printf("%d %d\n", bar.A, bar.B);
 return 0;
}

The C version prints what we expect:

$ gcc -fsanitize=undefined test.c && ./a.out
0 1
0 1

Your environment

I am on ArchLinux with Python 3.10.7. Python 3.11 and main are also affected. I also randomly tried Python 3.6 with the same result. (Python 3.6 is the oldest one that was easy to install.)

More comprehensive test case

Here's how I actually found the problem reported above. Using Hypothesis:

import ctypes
import string

from hypothesis import assume, example, given, note
from hypothesis import strategies as st

unsigned = [(ctypes.c_ushort, 16), (ctypes.c_uint, 32), (ctypes.c_ulonglong, 64)]
signed = [(ctypes.c_short, 16), (ctypes.c_int, 32), (ctypes.c_longlong, 64)]
types = unsigned + signed

unsigned_types = list(zip(*unsigned))[0]
signed_types = list(zip(*signed))[0]

names = st.lists(st.text(alphabet=string.ascii_letters, min_size=1), unique=True)


@st.composite
def fields_and_set(draw):
 names_ = draw(names)
 ops = []
 results = []
 for name in names_:
 t, l = draw(st.sampled_from(types))
 res = (name, t, draw(st.integers(min_value=1, max_value=l)))
 results.append(res)
 values = draw(st.lists(st.integers()))
 for value in values:
 ops.append((res, value))
 ops = draw(st.permutations(ops))
 return results, ops


def fit_in_bits(value, type_, size):
 expect = value % (2**size)
 if type_ not in unsigned_types:
 if expect >= 2 ** (size - 1):
 expect -= 2**size
 return expect


@given(fops=fields_and_set())
def test(fops):
 (fields, ops) = fops

 class BITS(ctypes.Structure):
 _fields_ = fields

 b = BITS()
 for (name, type_, size), value in ops:

 expect = fit_in_bits(value, type_, size)
 setattr(b, name, value)
 j = getattr(b, name)
 assert expect == j, f"{expect} != {j}"


if __name__ == "__main__":
 test()

Thanks to @mdickinson for pointing me in this direction.

Linked PRs

• gh-97588: Failing tests to demonstrate the issue #97589
• gh-97588: Fix ctypes structs #97702
• Backport 'Fix ctypes structs' (97702) to 3.11 #102774
• gh-97588: Move ctypes struct/union layout logic to Python #123352
• gh-97588: remove unused functions in _ctypes/cfield.c #124010

[matthiasgoergens]

Since I'm looking into bit-fields and ctypes anyway at the moment, I'll plan to work on a fix. But if someone else wants to give it a go, I'm happy to let them.

[matthiasgoergens]

As far as I can tell the problem might be with PyCField_FromDesc in cfield.c. The code looks both a bit complicated and is not expected to be a hot inner loop (it's only for setting up a Structure from a description.) So perhaps we might want to move most of its logic into Python.

Let's me dig some more.

[mdickinson]

@matthiasgoergens I think this is essentially the same issue as existing issues like #95496, #84039, #73939, #59324, etc; it would be good to try to consolidate some of these so we don't end up with an explosion of issues.

[matthiasgoergens]

@mdickinson Good idea. That's actually what I am already doing with the fix I have in the works, but consolidating the issues makes sense, too.

[matthiasgoergens]

@mdickinson I mentioned the issues that my PR fixes on the PR itself. Is there a way (or necessity) to also consolidate the issues themselves here?