Consider using Dask, Ray, etc.

Steps to replicate

df = pd.DataFrame([{'month': '2017-07-01'}, {'month': '2017-08-01'}, {'month': '2017-09-01'}])

import type_infer
type_infer.infer.type_check_date(df['month'])

Raises:

  File "/work/mindsdb/mindsdb_main/venv/lib/python3.10/site-packages/type_infer/infer.py", line 135, in type_check_date
    if dt.hour == 0 and dt.minute == 0 and dt.second == 0 and len(str(element)) <= 16:
  File "/work/mindsdb/mindsdb_main/venv/lib/python3.10/site-packages/pandas/core/generic.py", line 5989, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Series' object has no attribute 'hour'

Hi! Last year I had a great experience contributing to MindsDB's repos during the HacktoberFest.
This year I am not competing for MindsDB's own competition as I have less free time, but I was still looking for some cool and small issues I could help you with (and get that 2022 shirt :) ).

I was wondering if type_infer had any good first issues that could be labelled with the hacktoberfest label.

I have been writing a small typechecker for pandas + numpy + sklearn on my own, mainly to avoid losing my mind while refactoring some pipelines for Kaggle competitions, so this project looks right up my alley.

Stack trace:

RuntimeWarning: invalid value encountered in double_scalars
[45](...)
  randomness_per_index.append(S / np.log(N))

This issue is related to this one: mindsdb/mindsdb#3504

I wanted to improve the tests but wasnt sure about what was being tested or not, so I made a step in the CI to show de report so we can start to measture and improve the code coverage.

The current rule-based approach is failing to recognize a tag-filled column when it contains a large number of unique tags.

Additionally, it should be robust to additional separators or formatting, like brackets and quotes.

Right now, documentation is in a very preliminar and rough state.

Any help towards making it easier to understand the codebase and the purpose of the package will be of great help.

Important logic, we should unittest it with real-world examples and generated data

Should be quite a bit faster, hopefully installation too. Still, need to check behavior is equal (or very similar) to determine if the move is worth it or not.

We are currently hosting docs at https://mindsdb.github.io/type_infer, but this is not being communicated.

Should also be as lenient as possible with the versioning.

Column L1 in this dataset is inferred as short_text but should probably be reported as categorical instead.

Getting the following error when loading in context data for Writer Handler demo. My hypothesis is that is happening due to some columns containing many nulls which pandas handles as NaN.

python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

This will enable type inference on distributed setups, e.g. Sigmoid.

Related #54