Topic: llm-inference Goto Github

Some thing interesting about llm-inference

👇 Here are 302 public repositories matching this topic...

1b5d / llm-api

llm-inference,Run any Large Language Model behind a unified API

User: 1b5d

chatgpt gptq huggingface langchain llama llamacpp llm llm-inference machine-learning python

anarchy-ai / llm-vm

llm-inference,irresponsible innovation. Try now at https://chat.dev/

Organization: anarchy-ai

Home Page: https://anarchy.ai/

artificial-intelligence deep-learning distillation distillation-model llm llm-agent llm-inference llm-local llm-training machine-learning

b4rtaz / distributed-llama

llm-inference,Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

User: b4rtaz

distributed-computing llama2 llm llm-inference neural-network llms open-llm distributed-llm llama3

bentoml / openllm

llm-inference,Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

Organization: bentoml

Home Page: https://bentoml.com

llm llmops model-inference falcon fine-tuning stablelm llm-serving llama mpt vicuna

c0demunk33 / bespoke_automata

llm-inference,Bespoke Automata is a GUI and deployment pipline for making complex AI agents locally and offline

User: c0demunk33

agents ai automation chatbots developer-tools llm-inference

chenhunghan / ialacol

llm-inference,🪶 Lightweight OpenAI drop-in replacement for Kubernetes

User: chenhunghan

ai helm kubernetes langchain llm python openai cloudnative ggml gpu llamacpp cuda gptq llm-inference llm-serving

databricks / dbrx

llm-inference,Code examples and resources for DBRX, a large language model developed by Databricks

Organization: databricks

Home Page: https://www.databricks.com/

databricks gen-ai generative-ai llm llm-inference llm-training mosaic-ai

deftruth / awesome-llm-inference

llm-inference,📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

User: deftruth

Home Page: https://github.com/DefTruth/Awesome-LLM-Inference

flash-attention flash-attention-2 paged-attention streaming-llm streamingllm flash-decoding tensorrt-llm vllm awq awesome-llm

eastriverlee / llm.swift

llm-inference,LLM.swift is a simple, and readable library which lets you locally interact with LLMs with ease for macOS, iOS, visionOS, watchOS, and tvOS.

User: eastriverlee

ios llm llm-inference macos swift tvos visionos watchos gguf

efeslab / fiddler

llm-inference,Fast Inference of MoE Models with CPU-GPU Orchestration

Organization: efeslab

Home Page: https://arxiv.org/abs/2402.07033

llm llm-inference mixtral-8x7b mixture-of-experts local-inference

eulersearch / embedding_studio

llm-inference, Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

Organization: eulersearch

Home Page: https://embeddingstud.io/

embeddings embeddings-similarity fine-tuning llm-inference query-parser search-algorithm search-engine semantic-similarity unstructured-data unstructured-search

fasterdecoding / medusa

llm-inference,Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Organization: fasterdecoding

Home Page: https://sites.google.com/view/medusa-llm

llm llm-inference

flagai-open / aquila2

llm-inference,The official repo of Aquila2 series proposed by BAAI, including pretrained & chat large language models.

Organization: flagai-open

llm llm-inference llm-training

flashinfer-ai / flashinfer

llm-inference,FlashInfer: Kernel Library for LLM Serving

Organization: flashinfer-ai

Home Page: https://flashinfer.ai

flash-attention gpu large-large-models cuda pytorch tvm llm-inference

ghimiresunil / llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing

llm-inference,LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

User: ghimiresunil

bert huggingface large-language-models llm-inference llm-training open-source open-source-llm transformers llm-tutorials

hpcaitech / swiftinfer

llm-inference,Efficient AI Inference & Serving

Organization: hpcaitech

Home Page: https://hpc-ai.com/

artificial-intelligence deep-learning gpt inference llama llama2 llm-inference llm-serving

inferflow / inferflow

llm-inference,Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

User: inferflow

llama2 llamacpp llm-inference model-quantization multi-gpu-inference mixture-of-experts moe gemma falcon minicpm

intel / intel-extension-for-transformers

llm-inference,⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

Organization: intel

large-language-model chatbot 4-bits llm-inference llm-cpu chatpdf streamingllm intel-optimized-llamacpp speculative-decoding neural-chat

intel / neural-speed

llm-inference,An innovative library for efficient LLM inference via low-bit quantization

Organization: intel

Home Page: https://github.com/intel/neural-speed

cpu fp8 gaudi2 gpu int4 int8 llm-inference low-bit sparsity fp4 llm-fine-tuning mxformat nf4 llamacpp

internlm / lmdeploy

llm-inference,LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Organization: internlm

Home Page: https://lmdeploy.readthedocs.io/en/latest/

cuda-kernels deepspeed fastertransformer llm-inference turbomind internlm llama llm codellama llama2

kenza-ai / sagify

llm-inference,LLMs and Machine Learning done easily

Organization: kenza-ai

Home Page: https://kenza-ai.github.io/sagify/

sagemaker generative-ai large-language-model large-language-models llm llm-inference llmops openai ai-gateway anthropic

lean-dojo / leancopilot

llm-inference,LLMs as Copilots for Theorem Proving in Lean

Organization: lean-dojo

Home Page: https://leandojo.org

formal-mathematics lean lean4 llm-inference machine-learning theorem-proving

lightning-ai / litgpt

llm-inference,Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.

Organization: lightning-ai

Home Page: https://lightning.ai

ai artificial-intelligence deep-learning large-language-models llm llm-inference llms

liguodongiot / llm-action

llm-inference,本项目旨在分享大模型相关技术原理以及实战经验。

User: liguodongiot

Home Page: https://www.zhihu.com/column/c_1456193767213043713

llm llm-inference llm-serving llm-training llmops

liltom-eth / llama2-webui

llm-inference,Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

User: liltom-eth

llama-2 llama2 llm llm-inference

microsoft / aici

llm-inference,AICI: Prompts as (Wasm) Programs

Organization: microsoft

ai rust wasm wasmtime inference language-model llm llm-framework llm-inference llm-serving

microsoft / autogen

llm-inference,A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

Organization: microsoft

Home Page: https://microsoft.github.io/autogen/

agent-based-framework agent-oriented-programming chat chat-application chatbot chatgpt gpt gpt-35-turbo gpt-4 llm-agent

mistralai / mistral-src

llm-inference,Reference implementation of Mistral AI 7B v0.1 model.

Organization: mistralai

Home Page: https://mistral.ai/

llm llm-inference mistralai

morpheusais / morpheus

llm-inference,Morpheus - A Network For Powering Smart Agents - Compute + Code + Capital + Community

Organization: morpheusais

Home Page: https://mor.org/

agents ai compute ethereum llm-inference llms smart-contracts smart-agents

morpheuslord / hackbot

llm-inference,AI-powered cybersecurity chatbot designed to provide helpful and accurate answers to your cybersecurity-related queries and also do code analysis and scan analysis.

User: morpheuslord

ai chatbot cli-chat-app llama2 automation cybersecurity cybersecurity-education cybersecurity-tools llama2-7b llamacpp llm-inference runpod llama-api

neuralmagic / deepsparse

llm-inference,Sparsity-aware deep learning inference runtime for CPUs

Organization: neuralmagic

Home Page: https://neuralmagic.com/deepsparse/

machinelearning onnx inference computer-vision object-detection pruning quantization pretrained-models nlp cpus

nomic-ai / gpt4all

llm-inference,gpt4all: run open-source LLMs anywhere

Organization: nomic-ai

Home Page: https://gpt4all.io

llm-inference

nvidia / generativeaiexamples

llm-inference,Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Organization: nvidia

Home Page: https://nvidia.github.io/GenerativeAIExamples/latest/index.html

gpu-acceleration large-language-models llm llm-inference microservice nemo rag retrieval-augmented-generation tensorrt triton-inference-server

openvinotoolkit / openvino

llm-inference,OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Organization: openvinotoolkit

Home Page: https://docs.openvino.ai

inference deep-learning openvino ai computer-vision diffusion-models generative-ai llm-inference natural-language-processing nlp

predibase / lorax

llm-inference,Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Organization: predibase

Home Page: https://loraexchange.ai

fine-tuning gpt llama llm llm-inference llm-serving llmops lora model-serving pytorch

preternaturalai / mlx-swift-chat

llm-inference,A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.

Organization: preternaturalai

ios llm-inference macos mlx swiftui mlx-swift

promptslab / llmtuner

llm-inference,Tune LLM in few lines of code

Organization: promptslab

fine-tuning fine-tuning-llm finetune finetune-llm finetuning llm llm-framework llm-inference llm-training llmops

ray-project / ray-educational-materials

llm-inference,This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

Organization: ray-project

deep-learning distributed-machine-learning ray-distributed ray-tune ray ray-train ray-data ray-serve generative-ai llm

ray-project / ray-llm

llm-inference,RayLLM - LLMs on Ray

Organization: ray-project

Home Page: https://aviary.anyscale.com

distributed-systems large-language-models ray serving transformers llm llm-inference llm-serving llmops

rizerphe / local-llm-function-calling

llm-inference,A tool for generating function arguments and choosing what function to call with local LLMs

User: rizerphe

Home Page: https://local-llm-function-calling.readthedocs.io/

chatgpt-functions huggingface-transformers json-schema llm llm-inference openai-function-call openai-functions

rohan-paul / llm-finetuning-large-language-models

llm-inference,LLM (Large Language Model) FineTuning

User: rohan-paul

gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b

run-ai / genv

llm-inference,GPU environment and cluster management with LLM support

Organization: run-ai

Home Page: https://www.genv.dev

gpu docker gpus nvidia-gpu bash data-science deep-learning jupyter-notebook jupyterlab-extension vscode

safeailab / eagle

llm-inference,EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

Organization: safeailab

Home Page: https://arxiv.org/abs/2401.15077

large-language-models llm-inference speculative-decoding

sjtu-ipads / powerinfer

llm-inference,High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Organization: sjtu-ipads

bamboo-7b falcon large-language-models llama llm llm-inference local-inference

stoyan-stoyanov / llmflows

llm-inference,LLMFlows - Simple, Explicit and Transparent LLM Apps

User: stoyan-stoyanov

Home Page: https://llmflows.readthedocs.io

ai chatgpt gpt-4 llm llm-inference llmops llms machine-learning openai prompt-engineering python question-answering vector-database

substratusai / runbooks

llm-inference,Finetune LLMs on K8s by using Runbooks

Organization: substratusai

Home Page: https://www.substratus.ai

kubernetes mlops llmops kubernetes-operator llm-inference llm-serving llm-training ml-platform

llm-inference,🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.

Organization: superduperdb

Home Page: https://superduperdb.com

ai mlops torch transformers mongodb python pytorch ml database data