Describe the bug, including details regarding any error messages, version, and platform.
When building a dictionary array from Python values, pa.array silently ignores an unsigned index type and hands back the signed one instead:
import pyarrow as pa
dt = pa.dictionary(pa.uint32(), pa.string())
arr = pa.array(["a", "b", "a"], type=dt)
print(arr.type) # dictionary<values=string, indices=int32, ordered=0>
print(arr.type == dt) # False
The signed index types are honoured. The unsigned ones are all quietly downgraded to their signed counterpart:
| requested index type |
resulting index type |
| int8 / int16 / int32 / int64 |
same (honoured) |
| uint8 |
int8 |
| uint16 |
int16 |
| uint32 |
int32 |
| uint64 |
int64 |
Nothing warns or raises, so you end up with an array whose type is not the type you asked for.
The knock-on effect is more confusing than the original problem. pa.chunked_array takes a type argument too, and since the chunk it builds internally comes back with the signed index type, the type check fails with an error that never mentions dictionaries or index types:
>>> pa.chunked_array([["a", "b", "a"]], pa.dictionary(pa.uint32(), pa.string()))
ArrowTypeError: Array chunks must all be same type
The same call with a signed index type is fine:
>>> pa.chunked_array([["a", "b", "a"]], pa.dictionary(pa.int32(), pa.string())).type
dictionary<values=string, indices=int32, ordered=0>
And casting does honour unsigned index types, so they are clearly supported elsewhere:
>>> pa.chunked_array([["a", "b", "a"]]).cast(pa.dictionary(pa.uint32(), pa.string())).type
dictionary<values=string, indices=uint32, ordered=0>
I would expect pa.array(values, type=...) either to give back the index type that was asked for, or to raise if it cannot, rather than silently returning a different one. The chunked_array message could be clearer too, but that looks like a symptom rather than the cause.
This came up in narwhals, whose Categorical dtype maps to dictionary(uint32, string), so building a Categorical column from Python values fails on the PyArrow backend: narwhals-dev/narwhals#3775
Versions: pyarrow 23.0.1, Python 3.10.11, macOS arm64.
Component(s)
Python
Describe the bug, including details regarding any error messages, version, and platform.
When building a dictionary array from Python values,
pa.arraysilently ignores an unsigned index type and hands back the signed one instead:The signed index types are honoured. The unsigned ones are all quietly downgraded to their signed counterpart:
Nothing warns or raises, so you end up with an array whose type is not the type you asked for.
The knock-on effect is more confusing than the original problem.
pa.chunked_arraytakes a type argument too, and since the chunk it builds internally comes back with the signed index type, the type check fails with an error that never mentions dictionaries or index types:The same call with a signed index type is fine:
And casting does honour unsigned index types, so they are clearly supported elsewhere:
I would expect
pa.array(values, type=...)either to give back the index type that was asked for, or to raise if it cannot, rather than silently returning a different one. Thechunked_arraymessage could be clearer too, but that looks like a symptom rather than the cause.This came up in narwhals, whose
Categoricaldtype maps todictionary(uint32, string), so building a Categorical column from Python values fails on the PyArrow backend: narwhals-dev/narwhals#3775Versions: pyarrow 23.0.1, Python 3.10.11, macOS arm64.
Component(s)
Python