Thanks for your contribution and previous bug fixes.
I was able to train an Albert model from scratch.

But I am facing issues with using PyTorch-Lightning saved .ckpt file in HuggingFace code-base that I have.
I cannot use the .ckpt file directly, cause there is no corresponding index file present.(frompretrained
method throws error)
I am not entirely sure why pytorch-Lightning is saving model in ckpt format which is tf convention than the usual pytorch pt or bin

After downloading OpenWebText corpus, I extract it using the tar xvf openwebtext.tar.gz command. When I try running the python -m lmtuners.utils.tokenize_and_cache_data data/ data_tokenized_128/ --tokenizer_path bert-base-uncased-vocab.txt --max_length=64 command, I get an error saying

skipping urlsf_subset16-730_data.xz
0 tokens, 0 examples: 12% 2423/20610 [00:01<00:09, 1893.57it/s]'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte

for every file. Could you please help me overcome this issue? @shoarora

Hi,

I am trying to run the code in the repo with bert-base-multilingual-cased-vocab.txt vocab
But tokens being converted to ids when preprocessed are negative.
when passed to the network throws this error.

I tried ranging the max-length from 64 to 256.

RuntimeError: index out of range: Tried to access index -28619 out of table with 119546 rows. at /tmp/pip-req-build-ufslq_a9/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418