Support for current fairseq 0.10.2 about fastseq HOT 13 CLOSED

Hi @alaneckhardt @WQtong @gegallego @dearchill @ajar19 @0entr0py, FastSeq supports fairseq-0.10.x now! Please check it in the main branch and feel free to let us know if you meet any issue or have any feedback.

from fastseq.

@WQtong @alaneckhardt FYI, we are working on the support of fairseq-0.10.2.

from fastseq.

@alaneckhardt Supporting the latest version of fairseq is on our todo list. Currently we are collecting info from the community to set priorities.

from fastseq.

Thanks! I will wait for your decision.

from fastseq.

Hi Any updates on the plan? Currently if I downgrade the fairseq to 0.9.0 it throws error. If this tool does not support the newest fairse, it can not be used. Thank you.

from fastseq.

@WQtong Sorry about the issues. Our team is fully occupied by some other tasks (e.g., developing cuda kernels to speed up the inference). We may not be able to support the latest version of fairseq in a short period. However, the contributions from the community are very welcome!

from fastseq.

I tested that branch however the performance is even slower than the stock fairseq. The below code uses a base Transformer to translate a list of 30 sentences in 0.87 seconds while after adding "import fastseq" as the first line, the performance drops to 0.95 seconds. (this was tested on a Google Colab P100)

from fairseq.models.transformer import TransformerModel
import time

ja2en = TransformerModel.from_pretrained(
'./gdrive/MyDrive/japaneseModel/',
checkpoint_file='big.pretrain.pt',
data_name_or_path='./',
bpe='sentencepiece',
sentencepiece_model='./gdrive/MyDrive/japaneseModel/spm.en.nopretok.model',
is_gpu = True
)
ja2en.cuda()

from fastseq.

I tested that branch however the performance is even slower than the stock fairseq. The below code uses a base Transformer to translate a list of 30 sentences in 0.87 seconds while after adding "import fastseq" as the first line, the performance drops to 0.95 seconds. (this was tested on a Google Colab P100)

from fairseq.models.transformer import TransformerModel
import time

ja2en = TransformerModel.from_pretrained(
'./gdrive/MyDrive/japaneseModel/',
checkpoint_file='big.pretrain.pt',
data_name_or_path='./',
bpe='sentencepiece',
sentencepiece_model='./gdrive/MyDrive/japaneseModel/spm.en.nopretok.model',
is_gpu = True
)
ja2en.cuda()

@leminhyen2 Does your model run the forward process only once without beam search? If yes, it may be expected. Some details can be found here. If not, could you create an issue to describe more details for us to reproduce? and we can discuss more details there.

from fastseq.

@feihugis by forward process, you mean running the translate command like below? That is how I did it but with 30 sentences

listOfJapaneseSentences = ["こんにちは", "はい"]
resultList = ja2en.translate(listOfJapaneseSentences )
print(resultList)

How do I add beam search to this case? or is this only possible with command line of fairseq or fastseq?

from fastseq.

@leminhyen2 Sorry for the late response. Yes, you can find more examples here. Could you please create another issue to discuss your questions?

from fastseq.

I verified current fastseq works with fairseq 0.10.2., thanks a lot!

However I couldn't confirm the speedup - I tried our fairseq model trained on en-cs data. fastseq was slower for batch size 1 (3m43 vs 3m23), about the same for batch size 10 (1m28 vs 1m24) and very slightly faster for batch size 100 (0m36 vs 0m39).
I followed https://github.com/microsoft/fastseq/blob/main/examples/wmt/README.md#setting about how to run fastseq and compare it to fairseq and used pytorch 1.10.0 with CUDA 10.2. GPU was Tesla T4.

from fastseq.

Maybe if you can specify how the conda environment was created, I could pinpoint where the things went wrong.
Unfortunately, the link to WMT'16 en-de data from the examples doesn't work anymore, so I couldn't run it using your model.

from fastseq.

@alaneckhardt Could you please try a larger batch size (e.g., 256) as this table? We use the largest batch size to measure the performance. The reason is that FastSeq optimizes the gpu memory usage and operations to allow the larger batch size to improve the throughput. Currently the latency related issues are not our focus yet (but maybe in the future).

In terms of the environment setup, you can use this dockerfile (https://github.com/microsoft/fastseq/blob/main/docker/Dockerfile). The benchmarking script we used to measure the performance can be found here. You should be able to run it by cd fastseq/benchmarks && ./models/fs_wmt.sh.

The data link issue was resolved and it can be downloaded by following here.

from fastseq.