IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency¹.

• [2024/03] bigdl-llm has now become ipex-llm (see the migration guide here); you may find the original BigDL project here.
• [2024/02] ipex-llm now supports directly loading model from ModelScope (魔搭).
• [2024/02] ipex-llm added inital INT2 support (based on llama.cpp IQ2 mechanism), which makes it possible to run large-size LLM (e.g., Mixtral-8x7B) on Intel GPU with 16GB VRAM.
• [2024/02] Users can now use ipex-llm through Text-Generation-WebUI GUI.
• [2024/02] ipex-llm now supports Self-Speculative Decoding, which in practice brings ~30% speedup for FP16 and BF16 inference latency on Intel GPU and CPU respectively.
• [2024/02] ipex-llm now supports a comprehensive list of LLM finetuning on Intel GPU (including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA).
• [2024/01] Using ipex-llm QLoRA, we managed to finetune LLaMA2-7B in 21 minutes and LLaMA2-70B in 3.14 hours on 8 Intel Max 1550 GPU for Standford-Alpaca (see the blog here).

See the optimized performance of chatglm2-6b and llama-2-13b-chat models on 12th Gen Intel Core CPU and Intel Arc GPU below.

12th Gen Intel Core CPU		Intel Arc GPU
chatglm2-6b	llama-2-13b-chat	chatglm2-6b	llama-2-13b-chat

• Windows GPU: installing ipex-llm on Windows with Intel GPU
• Linux GPU: installing ipex-llm on Linux with Intel GPU
• Docker: using ipex-llm dockers on Intel CPU and GPU
• For more details, please refer to the installation guide

• llama.cpp: running ipex-llm for llama.cpp (using C++ interface of ipex-llm as an accelerated backend for llama.cpp on Intel GPU)
• vLLM: running ipex-llm in vLLM on both Intel GPU and CPU
• FastChat: running ipex-llm in FastChat serving on on both Intel GPU and CPU
• LangChain-Chatchat RAG: running ipex-llm in LangChain-Chatchat (Knowledge Base QA using RAG pipeline)
• Text-Generation-WebUI: running ipex-llm in oobabooga WebUI
• Benchmarking: running (latency and throughput) benchmarks for ipex-llm on Intel CPU and GPU

• Low bit inference
  • INT4 inference: INT4 LLM inference on Intel GPU and CPU
  • FP8/FP4 inference: FP8 and FP4 LLM inference on Intel GPU
  • INT8 inference: INT8 LLM inference on Intel GPU and CPU
  • INT2 inference: INT2 LLM inference (based on llama.cpp IQ2 mechanism) on Intel GPU
• FP16/BF16 inference
  • FP16 LLM inference on Intel GPU, with possible self-speculative decoding optimization
  • BF16 LLM inference on Intel CPU, with possible self-speculative decoding optimization
• Save and load
  • Low-bit models: saving and loading ipex-llm low-bit models
  • GGUF: directly loading GGUF models into ipex-llm
  • AWQ: directly loading AWQ models into ipex-llm
  • GPTQ: directly loading GPTQ models into ipex-llm
• Finetuning
  • LLM finetuning on Intel GPU, including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA
  • QLoRA finetuning on Intel CPU
• Integration with community libraries
  • HuggingFace tansformers
  • Standard PyTorch model
  • DeepSpeed-AutoTP
  • HuggingFace PEFT
  • HuggingFace TRL
  • LangChain
  • LlamaIndex
  • AutoGen
  • ModeScope
• Tutorials

For more details, please refer to the ipex-llm document website.

Over 50 models have been optimized/verified on ipex-llm, including LLaMA/LLaMA2, Mistral, Mixtral, Gemma, LLaVA, Whisper, ChatGLM2/ChatGLM3, Baichuan/Baichuan2, Qwen/Qwen-1.5, InternLM and more; see the list below.