Dataframe: access to dotted field names like SQL about lance HOT 5 CLOSED

Hmm if I could I would add a flatten parameter to to_pandas(), but we don't control that method (it's in PyArrow).

Other DataFrames do have decent support for nested columns, such as Polars. So I don't think flattening in general is what we want.

Perhaps we can provide a helpful snippet to teach them how to unflatten a column? IIRC it's just something like:

df.assign(nested = lambda df: [x['key'] for x in df['struct']])

from lance.

@pchalasani i think we can do this in LanceDB repo instead of the format level (please see the referencing PR)

from lance.

Nice, thanks, I seem to be conflating the two repos in my mind 😀

from lance.

For future reference for Lance users, you can write:

dataset.to_table(...).flatten().to_pandas()

If you have multiple levels of nested fields, you may need to call flatten() multiple times.

Maybe I can make this a tip in the user guide?

from lance.

from lance.