We now offer a pre-built Docker Image for the vLLM Worker that you can configure entirely with Environment Variables when creating the RunPod Serverless Endpoint:

• Required:

  • MODEL_NAME: Hugging Face Model Repository (e.g., openchat/openchat-3.5-1210).

• Optional:

  • MODEL_BASE_PATH: Model storage directory (default: /runpod-volume).
  • HF_TOKEN: Hugging Face token for private and gated models (e.g., Llama, Falcon).
  • NUM_GPU_SHARD: Number of GPUs to split the model across (default: 1).
  • QUANTIZATION: AWQ (awq) or SqueezeLLM (squeezellm) quantization.
  • MAX_CONCURRENCY: Max concurrent requests (default: 100).
  • DEFAULT_BATCH_SIZE: Token streaming batch size (default: 10). This reduces the number of HTTP calls, increasing speed 8-10x vs non-batching, matching non-streaming performance.
  • DISABLE_LOG_STATS: Enable (0) or disable (1) vLLM stats logging.
  • DISABLE_LOG_REQUESTS: Enable (0) or disable (1) request logging.

To build an image with the model baked in, you must specify the following docker arguments when building the image:

• Required
  • MODEL_NAME
• Optional
  • MODEL_BASE_PATH: Defaults to /runpod-volume for network storage. Use /models or for local container storage.
  • QUANTIZATION
  • HF_TOKEN
  • WORKER_CUDA_VERSION: 11.8 or 12.1 (default: 11.8 due to a small amount of workers not having CUDA 12.1 support yet. 12.1 is recommended for optimal performance).

sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg MODEL_BASE_PATH="/models" .

• LLaMA & LLaMA-2 (meta-llama/Llama-2-70b-hf, lmsys/vicuna-13b-v1.3, young-geng/koala, openlm-research/open_llama_13b, etc.)
• Mistral (mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.1, etc.)
• Mixtral (mistralai/Mixtral-8x7B-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1, etc.)
• Aquila & Aquila2 (BAAI/AquilaChat2-7B, BAAI/AquilaChat2-34B, BAAI/Aquila-7B, BAAI/AquilaChat-7B, etc.)
• Baichuan & Baichuan2 (baichuan-inc/Baichuan2-13B-Chat, baichuan-inc/Baichuan-7B, etc.)
• BLOOM (bigscience/bloom, bigscience/bloomz, etc.)
• ChatGLM (THUDM/chatglm2-6b, THUDM/chatglm3-6b, etc.)
• Falcon (tiiuae/falcon-7b, tiiuae/falcon-40b, tiiuae/falcon-rw-7b, etc.)
• GPT-2 (gpt2, gpt2-xl, etc.)
• GPT BigCode (bigcode/starcoder, bigcode/gpt_bigcode-santacoder, etc.)
• GPT-J (EleutherAI/gpt-j-6b, nomic-ai/gpt4all-j, etc.)
• GPT-NeoX (EleutherAI/gpt-neox-20b, databricks/dolly-v2-12b, stabilityai/stablelm-tuned-alpha-7b, etc.)
• InternLM (internlm/internlm-7b, internlm/internlm-chat-7b, etc.)
• MPT (mosaicml/mpt-7b, mosaicml/mpt-30b, etc.)
• OPT (facebook/opt-66b, facebook/opt-iml-max-30b, etc.)
• Phi (microsoft/phi-1_5, microsoft/phi-2, etc.)
• Qwen (Qwen/Qwen-7B, Qwen/Qwen-7B-Chat, etc.)
• Yi (01-ai/Yi-6B, 01-ai/Yi-34B, etc.)

And any other models supported by vLLM 0.2.6.

Ensure that you have Docker installed and properly set up before running the docker build commands. Once built, you can deploy this serverless worker in your desired environment with confidence that it will automatically scale based on demand. For further inquiries or assistance, feel free to contact our support team.

You may either use a prompt or a list of messages as input. If you use messages, the model's chat template will be applied to the messages automatically, so the model must have one. If you use prompt, you may optionally apply the model's chat template to the prompt by setting apply_chat_template to true.

Your list can contain any number of messages, and each message can have any role from the following list:

• user
• assistant
• system

The model's chat template will be applied to the messages automatically.

Example:

[
  {
    "role": "system",
    "content": "..."
  },
  {
    "role": "user",
    "content": "..."
  },
  {
    "role": "assistant",
    "content": "..."
  }
]