The corpus readers are currently grouped in a topic-ish fashion, e.g., lexicon projects are grouped together. But not all lexicon projects share the same structure, which creates all kinds of weird shenanigans when we try to load them. It also restricts us to arbitrary conventions about what information the user can extract from a certain corpus.
For example, we currently say that lexicon projects don't contain frequency fields, but some of them do, which is confusing.

It is probably better to just create a separate class for all these corpora, e.g., elp for English Lexicon Project, dlp for the Dutch Lexicon Project, etc.

This would cause us to have many more smaller classes, but all of these classes would be more useful than they are now.

Housekeeping is necessary

The feature extraction functions are currently all just functions. Making them objects would cause some serious reduction in overhead, and some more options for expansion in the future.

Currently, users have to do something like:

all_phonemes = get_characters(data, field='phonology')
features = extract_one_hot_phonemes(all_phonemes)
o = ONCTransformer(features)
X = o.fit_transform(data)

This isn't too bad, but can be simplified by making the extraction process above atomic:

all_phonemes = extract_one_hot_phonemes(data, field='phonology')

But this would require adding the same couple of lines of code to all extraction functions.

So, what I propose is:

o = ONCTransformer(OneHotPhonemeExtractor(), field='phonology')
X = o.fit_transform(data)

This merges the process of extracting the relevant phonemes from the data, and allows us to use inheritance for e.g. type checking, chaining etc.
I think it also puts less of a burden on the user, who no longer has to separately keep track of the features.

The readme is no longer valid.

I believe there is a small issue with dependencies as I met with an error about a module named ipapy which went along the lines of
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcd in position 1331: ordinal not in range(128)

I went and checked with the ipapy repo and it seemed that the issue was fixed in a recent pull request. is wordkit using an older version of ipapy?

Currently we treat frequencies as being on the same scale even though they might not be on the same scale. That is, if we currently combine two databases of frequency norms, and one of them was counted on the basis of 10M words, while the other was counted on the basis of 1M words, we will overestimate the frequency of the 10M corpus by a factor of 10.

Some solutions:

• Normalize the frequencies we get from corpora by the most frequent word.
• Manually look up the number of words on which the frequency norms were based, and use this to normalize.

The first option is straightforward, second one is a bit more difficult, especially because we might not know this for every source.

This would be a nice pip package, but only if it is kind of stable.
Currently, there's too much going on (I think) to put it on pip.

We currently offer a nice selection of corpora, but don't offer:

• MRC database
• Lexique DONE
• A spanish corpus DONE

Adding Lexique would especially be nice because this would give us a Celex-like database for French (e.g. including syllables and frequency counts)

Add typing, improve naming where necessary

Holographic features have been added, but they're not documented yet. Add the following:

• README
• docs
• papers

The word 'Transformer' has been used extensively throughout the module. This terminology is a cause of great confusion because the same term is used for the Transformer Model introduced in Attention is all you need which has been popular for various NLP applications in the past years. Though, that's not what the term means to refer to here.

The API of wordkit has changed somewhat.
Focus on the wordstore and the merging of corpora.

We currently use the sklearn.pipeline.FeatureUnion to combine different featurizers and corpora. This works great! But we want to replace it to add:

1. Merging different sources (e.g. merging the frequency table for a word from one data source with perceptual characteristics for the same word from another corpus).
2. Adding weights to transformers (e.g. assigning a weight of .5 to a phonology transformer to reduce the weight of phonology in any distance calculations).

The first point definitely makes a lot of sense and will be added ASAP, but I'm not sure about the second one.

When using the subtlexus (reader) function to load in the following corpus: SUBTLEXusfrequencyabove1.xls (obtained from here: http://crr.ugent.be/programs-data/subtitle-frequencies), I get the following error:

subtlexus('SUBTLEXusfrequencyabove1.xls', fields='orthography')

---

TypeError Traceback (most recent call last)
in
----> 1 subtlexus('SUBTLEXusfrequencyabove1.xls', fields='orthography')

~\anaconda3\lib\site-packages\wordkit\corpora\corpora\subtlex.py in subtlexus(path, fields)
46 def subtlexus(path,
47 fields=("orthography", "frequency")):
---> 48 return subtlex(path, fields, "eng-us")
49
50

~\anaconda3\lib\site-packages\wordkit\corpora\corpora\subtlex.py in subtlex(path, fields, language)
31 skiprows = 0
32
---> 33 return reader(path,
34 fields,
35 LANG2FIELD[language],

~\anaconda3\lib\site-packages\wordkit\corpora\base\reader.py in reader(path, fields, field_ids, language, preprocessors, opener, **kwargs)
68 field_ids = {}
69
---> 70 df = opener(path, **kwargs)
71 # Columns in dataset
72 colnames = set(df.columns)

~\anaconda3\lib\site-packages\wordkit\corpora\base\reader.py in _open(path, **kwargs)
27 extension = os.path.splitext(path)[-1]
28 if extension in {".xls", ".xlsx"}:
---> 29 df = pd.read_excel(path,
30 na_values=nans,
31 keep_default_na=False,

~\anaconda3\lib\site-packages\pandas\util_decorators.py in wrapper(*args, **kwargs)
294 )
295 warnings.warn(msg, FutureWarning, stacklevel=stacklevel)
--> 296 return func(*args, **kwargs)
297
298 return wrapper

TypeError: read_excel() got an unexpected keyword argument 'sep'

Attaching multiple IPA diacritics is currently supported for both vowels and consonants (excluding tone diacritics and click-related things). We currently can't featurize these unless we assume one-hot encoding or pre-defined features.

Ideally, we would use the IPA characteristics of each phoneme to add features, but this requires that we know in advance which types of diacritics we are going to encounter (this is not a huge problem). More problematic is that this kind of featurization leads to really big feature spaces.