BUG: pandas.read_json casts float column to int about pandas HOT 7 OPEN

Note that the behavior described above is different if the format is JSON-Lines and the "pyarrow" engine is used:

json_lines = b'{"col1": 1, "col2": 1.0}\n{"col1": 2, "col2": 2.0}'

df = pd.read_json(io.BytesIO(json_lines), lines=True, engine="pyarrow")
assert not (df["col1"].dtype == df["col2"].dtype)

On the other hand, the downcasting appears again if the "ujson" engine (the default one) is used:

json_lines = b'{"col1": 1, "col2": 1.0}\n{"col1": 2, "col2": 2.0}'

df = pd.read_json(io.BytesIO(json_lines), lines=True)
assert df["col1"].dtype == df["col2"].dtype

from pandas.

Thats a good point

from pandas.

Also note that this downcasting is not performed by pandas.read_csv:

csv_content = "col1,col2\n1,1.0\n2,2.0"

df = pd.read_csv(io.StringIO(csv_content))
assert not (df["col1"].dtype == df["col2"].dtype)

from pandas.

Additionally, str column is also cast to int:

d = [{"col1": 1, "col2": 1.0, "col3": "1"}, {"col1": 2, "col2": 2.0, "col3": "2"}]

df = pd.read_json(io.StringIO(json.dumps(d)))
assert df["col1"].dtype == df["col2"].dtype == df["col3"].dtype

from pandas.

Passing dtype=False, I get the expected behavior of the OP. But the docstring doesn't seem clear to me:

If True, infer dtypes; if a dict of column to dtype, then use those; if False, then don’t infer dtypes at all, applies only to the data.

Perhaps the language can be improved.

@albertvillanova - can you confirm if dtype=False satisfies your use-case? Labeling this as just a docs issue for now.

from pandas.

@rhshadrach thanks for your reply.

Unfortunately, passing dtype=False does not satisfy my use-case, because indeed I was passing dtype_backend="pyarrow" as well (I did not mention it in the description to make things simpler).

Therefore the float-to-int downcasting persists even if passing dtype=False when passing dtype_backend="pyarrow":

d = [{"col1": 1, "col2": 1.0}, {"col1": 2, "col2": 2.0}]

df = pd.read_json(io.StringIO(json.dumps(d)), dtype_backend="pyarrow")
assert df["col1"].dtype == df["col2"].dtype

df = pd.read_json(io.StringIO(json.dumps(d)), dtype_backend="pyarrow", dtype=False)
assert df["col1"].dtype == df["col2"].dtype

Additionally, I would like to ask if in the former case (when no passing dtype_backend="pyarrow"), there would be other side effects when passing dtype=False. Would other dtypes be treated differently?

from pandas.

Thanks @albertvillanova - I've reclassified this issue.

from pandas.