Repository for training an IP adapter for DINO
You can install the package using pip:
pip install git+https://github.com/isamu-isozaki/dino-ip-adapter-refiners.git
or you can use the rye
package manager:
rye sync
Download the dataset from the following link:
To download the images from urls, you can use the (img2dataset)[https://github.com/rom1504/img2dataset] library for efficient downloading.
img2dataset --url_list photo-concept-bucket/ --input_format "parquet" \
--url_col "url" --caption_col "cogvlm_caption" --output_format webdataset \
--output_folder photo-concept-bucket-webdataset --processes_count 16 --thread_count 64 --image_size 1024 \
--resize_only_if_bigger False --resize_mode="center_crop" --skip_reencode True
Create a .env file in the root of the project with the following variables:
PHOTO_CONCEPT=/path/to/photo-concept-bucket-webdataset
PHOTO_CONCEPT_PREENCODED=/save/path/
Because we're using webdatasets, you can save and load data from a remote location. We used Google Cloud Storage for this purpose.
Run the following command to precompute the features for the dataset:
python precompute_data.py --dataset photo_concept --start_shard 0 --end_shard 56 --batch_size 32
Batch size can be adjusted to fit the memory of the machine, 32 works for 40GB of memory.
If adding text encoder embeddings and working with mosaic you can encode with
python precompute_data.py --dataset="photo_concept" --start_shard=0 --end_shard=56 --batch_size=32 --encode_prompt --use_mosaic --compression=zstd
If using mosaic encoding, .mds is added to PHOTO_CONCEPT_PREENCODED
to .env add
TRAINING_PATH="/save/path"
but add .mds at the end if using mosaic.
To train the model, you can use the following command:
.venv/bin/python src/dino_ip_adapter_refiners
If using Mosaic, you first need to download index.json from PHOTO_CONCEPT_PREENCODED to the cache file location you specified. It will be along the lines of
gsutil cp GCP_SAVE_PATH/index.json CACHE_FOLDER_LOCATION