convolution is matmul so there should be a lora version for it.

im2col:

• A demo for LoRA on Convolution network(This repo)
• A network module for kohya_ss/sd-script
• An Extension for using this↑ in sd-webui

normally most of people train LoRA with kohya-ss/sd-scripts, (me too)

but lora only train green part, locon can train yellow part. Combine them can cover almost all of the layers in the model.

(I skip the porj in and porj out since they have very small channels, if you want to f/t them, maybe just f/t it without any trick.)

move locon folder into kohya-ss/sd-scripts and use

python3 sd-scripts/train_network.py \
  --network_module=locon.locon_kohya \
  --network_args="conv_dim=RANK_FOR_CONV" \
  --network_dim="RANK_FOR_LINEAR"

to train locon+lora for SD model

download Extension into sd-webui, and then use locon model as how you use lora model.

Onimai LoRA: https://huggingface.co/KBlueLeaf/onimai-locon-test

LoRA for Linear:

Convolution img2col:

LoRA for Convolution:

Another form:

[out_ch, in_ch*size**2] x [in_ch*size**2, out_h * out_w]
↓
[out_ch, LoRA_rank] x [LoRA_rank, in_ch*size**2] x [in_ch*size**2, out_h * out_w]
↓
[out_ch, LoRA_rank] x ([LoRA_rank, in_ch*size**2] x [in_ch*size**2, out_h * out_w])
↓
[out_ch, LoRA_rank] x [LoRA_rank, out_h * out_w]

FLOPS:

• before = out_ch * in_ch * size**2 * out_h * out_w

• after = out_ch * LoRA_rank * out_h * out_w + LoRA_rank * in_ch * size**2 * out_h * out_w

• after = (out_ch * LoRA_rank + LoRA_rank * in_ch * size**2) * out_h * out_w

Params to train:

• before = out_ch * in_ch * size**2

• after = LoRA_rank * in_ch * size**2 + LoRA_rank * out_ch