mseal / dx Goto Github PK

View Code? Open in Web Editor NEW

22.0 6.0 7.0 13.07 MB

Python support package for DEX visualization

Home Page: https://noteable-io.github.io/dx/

License: MIT License

Python 100.00%

pandas python datalink dex

dx's People

Contributors

Stargazers

Watchers

Forkers

willingc firmai-research datalayer-externals gaowudao datalearns danozworld lumenart

dx's Issues

support new sampling UI fields during resampling (numColumns, rowSamplingMethod, columnSamplingMethod)

add support for non-dataframe rendering

We're currently only capturing pd.DataFrame objects in the display formatters, but it would be nice to handle things like pd.Series, gdf.GeoDataFrame, and more.

Add conda-forge feedstock for the project

For now, PyPI is fine for releases. As we get further along, we'll want to add a feedstock to release on conda forge.

fix mkdocs snippet paths in docs

Almost all references seem to be broken -- https://noteable-io.github.io/dx/plotting/*_charts/ pages that pull from snippets in docs/_charts aren't appearing.

Direct links to https://noteable-io.github.io/dx/_charts/bar/ seem to work fine though.

May need to update based on https://facelessuser.github.io/pymdown-extensions/extensions/snippets/#specifying-snippet-locations

(Left: local mkdocs serve; right: https://noteable-io.github.io/dx/plotting/part_to_whole_charts/ -- both are pulling from plotting/basic_charts.md and similar sections in _charts)

https://github.com/noteable-io/dx/blob/main/docs/plotting/part_to_whole_charts.md?plain=1#L5

break up DEX models instead of subclassing

Gets rid of inherited properties that we may not care about for a majority of chart types.

handle shapely geometry objects

import geopandas as gpd
import numpy as np
import pandas as pd


df = pd.DataFrame(np.random.rand(20, 5))
gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df[0], df[1]))
gdf

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [2], in <cell line: 8>()
      6 df = pd.DataFrame(np.random.rand(20, 5))
      7 gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df[0], df[1]))
----> 8 gdf

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:262, in DisplayHook.__call__(self, result)
    260 self.start_displayhook()
    261 self.write_output_prompt()
--> 262 format_dict, md_dict = self.compute_format_data(result)
    263 self.update_user_ns(result)
    264 self.fill_exec_result(result)

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:151, in DisplayHook.compute_format_data(self, result)
    121 def compute_format_data(self, result):
    122     """Compute format data of the object to be displayed.
    123 
    124     The format data is a generalization of the :func:`repr` of an object.
   (...)
    149 
    150     """
--> 151     return self.shell.display_formatter.format(result)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/dataresource.py:51, in DXDataResourceDisplayFormatter.format(self, obj, **kwargs)
     49 display_id = str(uuid.uuid4())
     50 df_obj = pd.DataFrame(obj)
---> 51 payload, metadata = _render_dataresource(df_obj, display_id)
     52 # TODO: determine if/how we can pass payload/metadata with
     53 # display_id for the frontend to pick up properly
     54 return ({}, {})

File /opt/conda/lib/python3.9/site-packages/dx/formatters/dataresource.py:110, in _render_dataresource(df, display_id)
    106 # don't pass a dataframe in here, otherwise you'll get recursion errors
    107 with pd.option_context(
    108     "html.table_schema", dataresource_settings.DATARESOURCE_HTML_TABLE_SCHEMA
    109 ):
--> 110     ipydisplay(payload, raw=True, display_id=display_id)
    112 return (payload, metadata)

File /opt/conda/lib/python3.9/site-packages/IPython/core/display_functions.py:296, in display(include, exclude, metadata, transient, display_id, raw, clear, *objs, **kwargs)
    294 for obj in objs:
    295     if raw:
--> 296         publish_display_data(data=obj, metadata=metadata, **kwargs)
    297     else:
    298         format_dict, md_dict = format(obj, include=include, exclude=exclude)

File /opt/conda/lib/python3.9/site-packages/IPython/core/display_functions.py:93, in publish_display_data(data, metadata, source, transient, **kwargs)
     90 if transient:
     91     kwargs['transient'] = transient
---> 93 display_pub.publish(
     94     data=data,
     95     metadata=metadata,
     96     **kwargs
     97 )

File /opt/conda/lib/python3.9/site-packages/ipykernel/zmqshell.py:132, in ZMQDisplayPublisher.publish(self, data, metadata, source, transient, update)
    126 msg_type = 'update_display_data' if update else 'display_data'
    128 # Use 2-stage process to send a message,
    129 # in order to put it through the transform
    130 # hooks before potentially sending.
    131 msg = self.session.msg(
--> 132     msg_type, json_clean(content),
    133     parent=self.parent_header
    134 )
    136 # Each transform either returns a new
    137 # message or None. If None is returned,
    138 # the message has been 'used' and we return.
    139 for hook in self._hooks:

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in json_clean(obj)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in <listcomp>(.0)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:151, in json_clean(obj)
    148     return obj.strftime(ISO8601)
    150 # we don't understand it, it's probably an unserializable object
--> 151 raise ValueError("Can't clean for JSON: %r" % obj)

ValueError: Can't clean for JSON: <shapely.geometry.point.Point object at 0x7f689fe4a670>

generate DEX metadata alongside datalink metadata

should provide for closer integration with DEX views, to include prepopulating with display IDs and variable names

support custom renderable converters

settings.RENDERABLE_OBJECTS allows users to add other types that can be easily transformed into a dataframe by being wrapped in pd.DataFrame(____), but it would be nice to set more customized renderables that have dedicated conversion functions, like https://networkx.org/documentation/networkx-1.10/reference/generated/networkx.convert_matrix.to_pandas_dataframe.html

DX library outside of noteable.io

I am trying to use DX library outside of noteable.io notebook. eg. jupyter notebook or vscode. and the code runs there perfectly, but I dont see the output results like in noteable.io, Can someone please make it clear whether it is usable outside of noteable.io or not?
if its usable, then how? I'd like to have complete codes for the same.

also tell me, does DX have charting capabilities, is it similar to highcharts or Echarts? is DX capable of generating charts like matplotlib or seaborn? can it generate charts, bar graphs, etc. from the csv file that is given as input? is it capable of pure data visualization.

replace callout HTML with custom media type

dx logger exposes other loggers

https://noteables.slack.com/archives/C03U44DFPCZ/p1661183761622109

add setting for `noteable` attrs

Add setting to allow DEX metadata generation for .attrs with the noteable key.

Let's start using the GitHub Release feature

As we release things onto pypi, we should also update the releases in GitHub.

`import dx` is slow

`user_variable_name` can be `_1`, `_2`, etc

fix requirements-docs.txt usage

consolidate requirements-docs.txt and pyproject.toml configs to make it easier for building docs
possibly remove nox

remove optional ipython_shell args used everywhere

done in sidecar_comms already:

MSeal/sidecar_comms#34

fix `sampled_col_string_lengths` check

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:96, in handle_format(obj, with_ipython_display, ipython_shell, extra_metadata)
     95 try:
---> 96     payload, metadata = datalink_processing(
     97         obj,
     98         default_index_used,
     99         ipython_shell=ipython,
    100         with_ipython_display=with_ipython_display,
    101         extra_metadata=extra_metadata,
    102     )
    103 except Exception as e:

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:50, in datalink_processing(df, default_index_used, ipython_shell, with_ipython_display, extra_metadata)
     49 parent_display_id = determine_parent_display_id(dxdf)
---> 50 payload, metadata = format_output(
     51     dxdf.df,
     52     update=parent_display_id,
     53     display_id=dxdf.display_id,
     54     default_index_used=default_index_used,
     55     with_ipython_display=with_ipython_display,
     56     variable_name=dxdf.variable_name,
     57     extra_metadata=extra_metadata,
     58 )
     60 # this needs to happen after sending to the frontend
     61 # so the user doesn't wait as long for writing larger datasets

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:184, in format_output(df, update, display_id, default_index_used, with_ipython_display, variable_name, extra_metadata)
    183 for col, orig_length in orig_col_string_lengths.items():
--> 184     if orig_length > sampled_col_string_lengths[col]:
    185         truncated_string_columns.append(col)

KeyError: 'track.external_ids.isrc'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
Cell In[67], line 5
      3 audio_features_df.columns = "audio_feature." + audio_features_df.columns
      4 merged_track_df = track_df.merge(audio_features_df, left_on='track.uri', right_on='audio_feature.uri')
----> 5 merged_track_df.head()

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:268, in DisplayHook.__call__(self, result)
    266 self.start_displayhook()
    267 self.write_output_prompt()
--> 268 format_dict, md_dict = self.compute_format_data(result)
    269 self.update_user_ns(result)
    270 self.fill_exec_result(result)

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:157, in DisplayHook.compute_format_data(self, result)
    127 def compute_format_data(self, result):
    128     """Compute format data of the object to be displayed.
    129 
    130     The format data is a generalization of the :func:`repr` of an object.
   (...)
    155 
    156     """
--> 157     return self.shell.display_formatter.format(result)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:122, in DXDisplayFormatter.format(self, obj, **kwargs)
    119 def format(self, obj, **kwargs):
    121     if IN_NOTEBOOK_ENV and isinstance(obj, tuple(settings.RENDERABLE_OBJECTS)):
--> 122         handle_format(obj)
    123         return ({}, {})
    125     return DEFAULT_IPYTHON_DISPLAY_FORMATTER.format(obj, **kwargs)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:106, in handle_format(obj, with_ipython_display, ipython_shell, extra_metadata)
    104     logger.debug(f"Error in datalink_processing: {e}")
    105     # fall back to default processing
--> 106     payload, metadata = format_output(
    107         obj,
    108         default_index_used=default_index_used,
    109         with_ipython_display=with_ipython_display,
    110         extra_metadata=extra_metadata,
    111     )
    113 return payload, metadata

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:184, in format_output(df, update, display_id, default_index_used, with_ipython_display, variable_name, extra_metadata)
    182 truncated_string_columns = []
    183 for col, orig_length in orig_col_string_lengths.items():
--> 184     if orig_length > sampled_col_string_lengths[col]:
    185         truncated_string_columns.append(col)
    187 payload = generate_body(
    188     df,
    189     display_id=display_id,
    190     default_index_used=default_index_used,
    191 )

KeyError: 'track.external_ids.isrc'

datasets with `pd.NA` fail JSON serializing during ipython display

To reproduce:

df = pd.DataFrame(
    [
        [1, pd.NA],
        [2, 3],
    ],
    columns=['foo', 'bar']
)
dx.display(df)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [69], in <cell line: 8>()
      1 df = pd.DataFrame(
      2     [
      3         [1, pd.NA],
   (...)
      6     columns=['foo', 'bar']
      7 )
----> 8 df

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:266, in DisplayHook.__call__(self, result)
    264 self.fill_exec_result(result)
    265 if format_dict:
--> 266     self.write_format_data(format_dict, md_dict)
    267     self.log_output(format_dict)
    268 self.finish_displayhook()

File /opt/conda/lib/python3.9/site-packages/ipykernel/displayhook.py:71, in ZMQShellDisplayHook.write_format_data(self, format_dict, md_dict)
     70 def write_format_data(self, format_dict, md_dict=None):
---> 71     self.msg['content']['data'] = json_clean(encode_images(format_dict))
     72     self.msg['content']['metadata'] = md_dict

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:145, in json_clean(obj)
    143     out = {}
    144     for k,v in obj.items():
--> 145         out[str(k)] = json_clean(v)
    146     return out
    147 if isinstance(obj, datetime):

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in json_clean(obj)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in <listcomp>(.0)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in json_clean(obj)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:131, in <listcomp>(.0)
    128     obj = list(obj)
    130 if isinstance(obj, list):
--> 131     return [json_clean(x) for x in obj]
    133 if isinstance(obj, dict):
    134     # First, validate that the dict won't lose data in conversion due to
    135     # key collisions after stringification.  This can happen with keys like
    136     # True and 'true' or 1 and '1', which collide in JSON.
    137     nkeys = len(obj)

File /opt/conda/lib/python3.9/site-packages/ipykernel/jsonutil.py:151, in json_clean(obj)
    148     return obj.strftime(ISO8601)
    150 # we don't understand it, it's probably an unserializable object
--> 151 raise ValueError("Can't clean for JSON: %r" % obj)

ValueError: Can't clean for JSON: <NA>

can't render groupby+resample dataframes

If we have a dataframe with a MultiIndex where one of the levels' names is already in the dataframe columns, the final .reset_index() before passing to the frontend will break with a ValueError: cannot insert keyword_column, already exists. Probably need to remove columns that are in the MultiIndex to preserve the index structure.

rename `default` display mode to `plain` for clarity

support `DataFrame.attrs` to update DEX rendering metadata

Populating a dataframe's .attrs with arbitrary metadata should be captured and parsed during handle_format() to allow predefined DEX rendering defaults (title, subtitle, view type, etc).

Should be based off of the original DEX metadata modeling in #87, but with a translation layer to capture .attrs and add to new/existing views.

replace globals with custom `DXDataFrame` helper class

DXDataFrame:

handle df conversions for id, display ID, hash, columns
keep track of column transformations

log a warning when resampling a stale display id

We can get into a confusing state if a resample request is received with an untracked display id (likely from kernel session restart) - at a minimum we should log the behavior

default index shouldn't be string type

https://noteables.slack.com/archives/C03U44DFPCZ/p1661184241535149

Allow dx.disable to turn off pandas display

Running dx.disable() should run this:

pd.set_option("display.html.table_schema", False)

handle `np.ndarray` tracking

resampling on some date string formats returns 0 rows

difference in formatting stored in duckdb versus what's sent to the frontend

update docs to swap `make_dashboard` for `dashboard`

Wording & screenshots.

move pieces of renaming+assignment comms out of this package

Should go in a different package for any features that aren't directly involved with the dataframe formatting/filtering.

handle date/timestamp values

may need to coordinate with frontend - these are currently turned into either j1970/epoch time as a float (with float metrics), or as strings (and no metrics are returned)

`groupby` operations returning pd.Series objects raise `ValueError: cannot insert <column>, already exists`

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:87, in handle_format(obj, ipython_shell)
     86 try:
---> 87     payload, metadata = datalink_processing(
     88         obj,
     89         default_index_used,
     90         ipython_shell=ipython,
     91     )
     92 except Exception as e:

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:50, in datalink_processing(df, default_index_used, ipython_shell)
     48     logger.debug(f"df is subset of existing {parent_display_id=}")
---> 50 payload, metadata = format_output(
     51     dxdf.df,
     52     update=parent_display_id,
     53     display_id=dxdf.display_id,
     54     has_default_index=default_index_used,
     55 )
     57 # this needs to happen after sending to the frontend
     58 # so the user doesn't wait as long for writing larger datasets

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:158, in format_output(df, update, display_id, has_default_index)
    156 sampled_df_dimensions = get_df_dimensions(df, prefix="truncated")
--> 158 payload = generate_body(df, display_id=display_id)
    160 dataframe_info = {
    161     "default_index_used": has_default_index,
    162     **orig_df_dimensions,
    163     **sampled_df_dimensions,
    164 }

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:133, in generate_body(df, display_id)
    132 if settings.DISPLAY_MODE == DXDisplayMode.simple:
--> 133     data = clean_df.reset_index().to_dict("records")
    134 elif settings.DISPLAY_MODE == DXDisplayMode.enhanced:

File /opt/conda/lib/python3.9/site-packages/pandas/util/_decorators.py:311, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    306     warnings.warn(
    307         msg.format(arguments=arguments),
    308         FutureWarning,
    309         stacklevel=stacklevel,
    310     )
--> 311 return func(*args, **kwargs)

File /opt/conda/lib/python3.9/site-packages/pandas/core/frame.py:5848, in DataFrame.reset_index(self, level, drop, inplace, col_level, col_fill)
   5844             level_values = algorithms.take(
   5845                 level_values, lab, allow_fill=True, fill_value=lev._na_value
   5846             )
-> 5848         new_obj.insert(0, name, level_values)
   5850 new_obj.index = new_index

File /opt/conda/lib/python3.9/site-packages/pandas/core/frame.py:4443, in DataFrame.insert(self, loc, column, value, allow_duplicates)
   4441 if not allow_duplicates and column in self.columns:
   4442     # Should this be a different kind of error??
-> 4443     raise ValueError(f"cannot insert {column}, already exists")
   4444 if not isinstance(loc, int):

ValueError: cannot insert Location Description, already exists

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In [9], line 1
----> 1 df.groupby(['IUCR', 'Primary Type'])['Location Description'].value_counts()

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:262, in DisplayHook.__call__(self, result)
    260 self.start_displayhook()
    261 self.write_output_prompt()
--> 262 format_dict, md_dict = self.compute_format_data(result)
    263 self.update_user_ns(result)
    264 self.fill_exec_result(result)

File /opt/conda/lib/python3.9/site-packages/IPython/core/displayhook.py:151, in DisplayHook.compute_format_data(self, result)
    121 def compute_format_data(self, result):
    122     """Compute format data of the object to be displayed.
    123 
    124     The format data is a generalization of the :func:`repr` of an object.
   (...)
    149 
    150     """
--> 151     return self.shell.display_formatter.format(result)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:106, in DXDisplayFormatter.format(self, obj, **kwargs)
    103 def format(self, obj, **kwargs):
    105     if IN_NOTEBOOK_ENV and isinstance(obj, tuple(settings.RENDERABLE_OBJECTS)):
--> 106         handle_format(obj)
    107         return ({}, {})
    109     return DEFAULT_IPYTHON_DISPLAY_FORMATTER.format(obj, **kwargs)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:95, in handle_format(obj, ipython_shell)
     93     logger.debug(f"Error in datalink_processing: {e}")
     94     # fall back to default processing
---> 95     payload, metadata = format_output(obj, has_default_index=default_index_used)
     97 return payload, metadata

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:158, in format_output(df, update, display_id, has_default_index)
    155 df = sample_if_too_big(df, display_id=display_id)
    156 sampled_df_dimensions = get_df_dimensions(df, prefix="truncated")
--> 158 payload = generate_body(df, display_id=display_id)
    160 dataframe_info = {
    161     "default_index_used": has_default_index,
    162     **orig_df_dimensions,
    163     **sampled_df_dimensions,
    164 }
    165 metadata = generate_metadata(display_id=display_id, **dataframe_info)

File /opt/conda/lib/python3.9/site-packages/dx/formatters/main.py:133, in generate_body(df, display_id)
    130 clean_df = df.astype(object).where(df.notnull(), None)
    132 if settings.DISPLAY_MODE == DXDisplayMode.simple:
--> 133     data = clean_df.reset_index().to_dict("records")
    134 elif settings.DISPLAY_MODE == DXDisplayMode.enhanced:
    135     data = clean_df.reset_index().transpose().values.tolist()

File /opt/conda/lib/python3.9/site-packages/pandas/util/_decorators.py:311, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    305 if len(args) > num_allow_args:
    306     warnings.warn(
    307         msg.format(arguments=arguments),
    308         FutureWarning,
    309         stacklevel=stacklevel,
    310     )
--> 311 return func(*args, **kwargs)

File /opt/conda/lib/python3.9/site-packages/pandas/core/frame.py:5848, in DataFrame.reset_index(self, level, drop, inplace, col_level, col_fill)
   5842         if lab is not None:
   5843             # if we have the codes, extract the values with a mask
   5844             level_values = algorithms.take(
   5845                 level_values, lab, allow_fill=True, fill_value=lev._na_value
   5846             )
-> 5848         new_obj.insert(0, name, level_values)
   5850 new_obj.index = new_index
   5851 if not inplace:

File /opt/conda/lib/python3.9/site-packages/pandas/core/frame.py:4443, in DataFrame.insert(self, loc, column, value, allow_duplicates)
   4437     raise ValueError(
   4438         "Cannot specify 'allow_duplicates=True' when "
   4439         "'self.flags.allows_duplicate_labels' is False."
   4440     )
   4441 if not allow_duplicates and column in self.columns:
   4442     # Should this be a different kind of error??
-> 4443     raise ValueError(f"cannot insert {column}, already exists")
   4444 if not isinstance(loc, int):
   4445     raise TypeError("loc must be int")

ValueError: cannot insert Location Description, already exists

No issues with groupby operations that return dataframes though:

non-pandas DataFrame equality checks don't pass when getting the df variable name

preprocess `datetime.date` to pd.Timestamp/datetime values instead of strings

dx cannot be installed on Python 3.11

Describe the bug
Our company installed the newest Python version lately, namely 3.11. I found that it is not possible for me to install dx:

dx has PyArrow as requirement. PyArrow does currently not support Python 3.11 in its latest stable version (but within its nightly build, to be installed with pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --prefer-binary --pre pyarrow). Therefore, I installed the pre-release version (which supports 3.11) in the hope to get dx installed, but discovered that dx has an upper bound for the version of pyarrow (the pyarrow<9.0.0 part in the following output listing) as requirement.

C:\Users\peter.jung>pip install dx --upgrade
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: dx in c:\users\peter.jung\appdata\roaming\python\python311\site-packages (0.1.22)
Collecting dx
  Downloading dx-1.2.0-py3-none-any.whl (34 kB)
Collecting SQLAlchemy<2.0.0,>=1.4.40
  Downloading SQLAlchemy-1.4.45-cp311-cp311-win_amd64.whl (1.6 MB)
     ---------------------------------------- 1.6/1.6 MB 791.0 kB/s eta 0:00:00
Requirement already satisfied: ipython>=7.31.1 in c:\users\peter.jung\appdata\roaming\python\python311\site-packages (from dx) (8.7.0)
Requirement already satisfied: pandas<2.0.0,>=1.3.5 in c:\users\peter.jung\appdata\roaming\python\python311\site-packages (from dx) (1.5.2)
Collecting pyarrow<9.0.0,>=8.0.0
  Downloading pyarrow-8.0.0.tar.gz (846 kB)
     ---------------------------------------- 846.6/846.6 kB 1.5 MB/s eta 0:00:00
  Installing build dependencies ... done

(PyArrow 8.0.0 is bad since it gives a "missing cmake" error which was fixed in the nightly build.)

I propose to remove this upper bound and adapt dx to pyarrow's new version. Otherwise, we will always have something like the source code equivalent of a festering wound.

Expected behavior
No problems to install dx on Python 3.11. A necessary (and sufficient) condition would be to remove the upper bound for pyarrow's version.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

OS: Windows
Python Version: 3.11
Library Version: 1.2.0