BUG: Series stealing references from CategoricalIndex is invalid for read-only arrays

[vyasr]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Not necessary for pandas 3.0, but I validated this in both the 3.0 rc and 2.3.3
# and this setting makes the MRE work for both.
pd.set_option("mode.copy_on_write", True)

# We must use an int8 array here or pandas will make a (writeable) copy of the array in
# https://github.com/pandas-dev/pandas/blob/499c5d4dd52a8645bf96c39bad60613097e84c06/pandas/core/dtypes/cast.py#L878
# We also must convert the codes to a numpy array since that produces a read-only array,
# whereas pandas has more internal logic to handle an Index correctly in CoW mode.
codes = pd.Index([0, 1, 2, 3], dtype="int8").to_numpy()
cats = pd.Index(["a", "b", "c", "d"])
data = pd.Categorical.from_codes(codes, cats)

# We can't create a series directly from the Categorical data because the
# implementation details in pandas prevent copies in this case. When we construct a
# Series from a CategoricalIndex pandas tries to steal references to optimize
# copying in CoW mode, which is necessary to observe the error.
s = pd.Series(pd.Index(data))
s[[False, False, True, True]] = cats[2:4]

Issue Description

The above example will fail with an error

 File "${SITE}/pandas/core/arrays/_mixins.py", line 269, in __setitem__
 self._ndarray[key] = value
 ~~~~~~~~~~~~~^^^^^
ValueError: assignment destination is read-only

The issue arises under the following circumstances:

1. mode.copy_on_write is enabled
2. A Series (or column in a DataFrame) is constructed from a read-only array
3. The Series is constructed from an input that is determined not to have any other outstanding references such that CoW will not force a copy the first time an operation occurs.

Under these circumstances, pandas will not currently realize that the input data is read-only and will attempt to modify it, resulting in the above error. This example is a fairly specific case where this occurs, but I suspect that there are other similar cases where it is possible to end up with a read-only array inside a pandas object in CoW mode. The challenge is that such cases are easily obscured by any references floating around. Point 3 above is particularly delicate. While debugging this issue in my original example it took a lot of work to distill it into a minimal example because the reference counting logic in pandas will result in copies at various points in CoW mode if any foreign references exist, and I found it quite easy to wind up in cases where such references were preserved in orphaned reference cycles or hidden in other variables. In such cases pandas will defensively make copies that would cover up issues with a read-only input array.

Expected Behavior

When input data is read-only, in CoW mode pandas should check that blocks were constructed from a read-only input and make a copy when writing if necessary. That check could probably be inserted around here in setitem, but I don't know if that is the best place for it.

Installed Versions

Details

❯ python
Python 3.13.11 | packaged by conda-forge | (main, Dec 6 2025, 11:24:03) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit : 1a3230dc5be4c87b8356765ea3b6568d37cb82fd
python : 3.13.11
python-bits : 64
OS : Linux
OS-release : 5.4.0-208-generic
Version : #228-Ubuntu SMP Fri Feb 7 19:41:33 UTC 2025
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0rc0
numpy : 2.4.0rc1
dateutil : 2.9.0.post0
pip : 25.3
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
psycopg2 : None
pymysql : None
pyarrow : None
pyiceberg : None
pyreadstat : None
pytest : None
python-calamine : None
pytz : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
qtpy : None
pyqt5 : None

[mroeschke]

Just clarifying that this probably applies to a Series constructed by any Index backed by read-only data. On main

In [1]: import pandas as pd
 ...: import numpy as np
 ...: 
 ...: arr = np.array([0, 1])
 ...: arr.flags.writeable = False
 ...: ser = pd.Series(pd.Index(arr))
 ...: ser[[False, True]] = [0, 1]
...
File ~/pandas/core/array_algos/putmask.py:95, in putmask_without_repeat(values, mask, new)
 93 # i.e. values[mask] = new
 94 elif mask.shape[-1] == shape[-1] or shape[-1] == 1:
---> 95 np.putmask(values, mask, new)
 96 else:
 97 raise ValueError("cannot assign mismatch length to masked array")

ValueError: putmask: output array is read-only

In [1]: import pandas as pd
 ...: import numpy as np
 ...: 
 ...: arr = np.array([0, 1])
 ...: arr.flags.writeable = False
 ...: idx = pd.Index(arr) # reference still alive
 ...: ser = pd.Series(idx)
 ...: ser[[False, True]] = [0, 2]

In [2]: ser
Out[2]: 
0 0
1 2
dtype: int64

The inconsistency is a bug IMO. I don't recall whether pandas should respect the read-only input during write operations and raise generally or deem it "surprising for users that they cannot write" and copy in this case

[jorisvandenbossche]

And this will happen for any Series where the underlying array is read-only, i.e. this is #62144

It is that with CoW this becomes more common. I had thought of ser.values in the issue, but going through Index is at the moment also a way to get this.

IMO, as I argued in that issue, a setitem operation on a pandas Series or DataFrame object should essentially always work, and so I would consider that as a bug that we don't take into account the case of read-only arrays (you can also get that without CoW, like some corner cases with pyarrow).

But let's continue the discussion of that part in #62144 ?

---

That said, there is indeed another issue here. The inconsistency boils down to: pd.Index(..) has a copy=False default, while in pd.Series() we decided to copy by default if the input is an array (but use the CoW tracking for pandas Index/Series input).

As a result, constructing the series directly does not give an underlying read-only array (it will copy the categorical or array):

>>> arr = np.array([0, 1])
... arr.flags.writeable = False
>>> ser = pd.Series(arr)
>>> ser[[False, True]] = 100
>>> ser
0 0
1 100
dtype: int64

While constructing the Series through a temporary Index causes the array to not be copied, but the reference tracking is lost because the Index was only temporary.

So I would say that the inconsistency is caused by Index not copying the input data by default. We changed Series to copy array input by default when we introduced CoW. We should maybe do the same for Index?