These one click templates allow you to quickly boot up an API for a given language model.

• Make sure to read through the README file on the templates!
• Vast.AI is recommended for lowest cost per hour with smaller GPUs like A4000 and A2000. Runpod is recommended (better user interface) if using larger GPUs like A6000, A100 or H100.

Advanced inferencing scripts (incl. for function calling) are available for purchase here.

To support the Trelis Research YouTube channel, you can sign up for an account with this affiliate link.

Text Generation Inference (fastest):

• Mistral 7B api

vLLM:

• Mistral 7B v0.2 AWQ
• Post a new issue if you would like other templates

To support the Trelis Research YouTube channel, you can sign up for an account with this link.

Text Generation Inference (fastest):

• OpenChat 3.5 7B API.
• Mixtral Instruct API. Requires an A6000 or A100 or H100.
• Llama 70B API by TrelisResearch.
• Deepseek Coder 33B Template.
• Medusa Vicuna (high speed speculative decoding - mostly a glamour template because OpenChat with AWQ is better quality and faster)

vLLM One-click templates:

• Mistral 7B AWQ

llama.cpp One-click templates (new, faster than TGI):

• Mistral 7B Instruct v0.2 8-bit

Post a new issue if you would like other templates

One-click templates for function-calling are located on the HuggingFace model cards. Check out the collection here.