Note: This script was adapted from HuggingFace's Transformers/language-modeling code.

bittensor must be installed either locally or in the virtual environment you are working from.

Run pip install -r requirements.txt to install the additional packages for this script.

Fine-tuning the library models for language modeling on a text dataset for GPT, GPT-2. Causal languages like this are trained or fine-tuned using a causal language modeling (CLM) loss.

The following examples, we will run on datasets hosted on Bittensor's IPFS mountain dataset, on HuggingFace's dataset hub or with your own text files.

By default, the script will fine-tune GPT2 for bittensor's mountain dataset. Running:

python finetune_using_clm.py

will tune gpt2 with bittensor's dataset and save the output to tuned-model.

to change the model you are tuning to, e.g. distilgpt2, run:

python finetune_using_clm.py model.name=distilgpt2

Some sample models to try are available under the server customization section of bittensor's documentation. A full list of models that can be trained by this script are available on huggingface.

Any text dataset on huggingface should work by default by overriding the dataset.name and dataset.config parameters:

python finetune_using_clm.py dataset.name=wikitext dataset.config_name=wikitext-103-v1

If you have a .txt file saved locally, you can override dataset.name as above:

python finetune_using_clm.py dataset.name=./path/to/your/data.txt

Note if using your own data, you may have many short sentences and the block size may be insufficient for reasonable performance. It's recommended you pass the flag dataset.concatenate_raw=true to give the model more context when training. This will reduce the number of batches.

All configurable parameters are visible and documented in conf/config.yaml. The defaults are chosen for quick training and not tuned; you will need to experiment and adjust these.

To serve your tuned model on bittensor, just override neuron.model_name with the path to your tuned model:

btcli run ..... --neuron.model_name=/home/user/models/my-tuned-gpt2

Early stopping is not yet supported. Many features are implemented but not thoroughly tested, if you encounter an issue, reach out on discord or (preferably) create an issue on this github page.