jonasmaertens / bpevectorizer Goto Github PK

This package contains a fast one-hot encoding vectorizer with youtokentome byte-pair-encoding.

Build the package

pip install build cython
python -m build

Install the wheel from the dist directory

pip install dist/bpe_vectorizer-{something}.whl

Or install the package directly from the .tar.gz in the release

pip install cython
pip install bpe_vectorizer-{something}.tar.gz

Import BPEVectorizer like this:

from bpe_vectorizer import BPEVectorizer

Usage of the vectorizer:

vectorizer = BPEVectorizer.train("train.csv", model="bpe.yttm", vocab_size=10000)
x_train = vectorizer.transform(train_df["tokens"])
x_test = vectorizer.transform(test_df["tokens"])

Note that BPEVectorizer cannot be pickled properly (still depends on the absolute path of the model file). Instead, use the .yttm file that is created during training and load it like this:

vectorizer = BPEVectorizer.load("bpe.yttm")