richarddwang / hugdatafast Goto Github PK

the doc says

This will install also the lastest fastai and nlp.

however, when I have tested to use hugdatafast, I have ended up install fastai myself via pip install fastai --upgrade.

the following import statement
from hugdatafast import *

it complained below

ModuleNotFoundError: No module named 'fastai.text.all'

cola_val = tokenized_cola['validation']
lm_dataset = LMTransform(cola_val, max_len=20, text_col='text_idxs').map()

TypeError Traceback (most recent call last)
in
1 cola_val = tokenized_cola['validation']
----> 2 lm_dataset = LMTransform(cola_val, max_len=20, text_col='text_idxs').map()
3
4 print('Original dataset:')
5 print('num of samples:', len(cola['validation']))

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/hugdatafast/transform.py in map(self, split_kwargs, cache_dir, cache_name, **kwargs)
56 kwargs.update(split_kwargs[split])
57 if hasattr(kwargs, 'remove_columns'): self._check_outcols(kwargs['remove_columns'], split)
---> 58 new_dsets[split] = self._map(dset, split, **kwargs)
59 # return
60 if self.single: return new_dsets['Single']

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/hugdatafast/transform.py in _map(self, hf_dset, split, batch_size, **kwargs)
215 output_schema = self.get_output_schema(hf_dset, kwargs.pop('test_batch_size', 20))
216 return hf_dset.map(function=self, batched=True, batch_size=batch_size, with_indices=True,
--> 217 arrow_schema=output_schema, **kwargs)
218
219 def get_output_schema(self, hf_dset, test_batch_size=20):

TypeError: map() got an unexpected keyword argument 'arrow_schema'

the following code snippets

dls = HF_Datasets(tokenized_datasets, cols=['text_idxs', 'label'], hf_toker=hf_tokenizer)

generates an error below

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-c30dd56e3ec8> in <module>()
      1 tokenized_datasets = datasets.map(simple_tokenize_func({'sentence':'text_idxs'}, hf_tokenizer))
----> 2 dls = HF_Datasets(tokenized_datasets, cols=['text_idxs', 'label'], hf_toker=hf_tokenizer)

2 frames
/usr/local/lib/python3.6/dist-packages/fastcore/utils.py in store_attr(names, self, but, **attrs)
     95     args,varargs,keyw,locs = inspect.getargvalues(fr)
     96     if self is None: self = locs[args[0]]
---> 97     if not hasattr(self, '__stored_args__'): self.__stored_args__ = {}
     98     if attrs: return _store_attr(self, **attrs)
     99 

AttributeError: 'str' object has no attribute '__stored_args__'

seems like fastcore has got upgraded?