hypertseng Goto Github PK

awesome-llm-inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

cuda-learn-note

🎉CUDA 笔记 / 大模型手撕CUDA / C++笔记，更新随缘: flash_attn、sgemm、sgemv、warp reduce、block reduce、dot product、elementwise、softmax、layernorm、rmsnorm、hist etc.

cuda-programming

Sample codes for my CUDA programming book

cutlass

CUDA Templates for Linear Algebra Subroutines

falcon-rw-1b_mindspore

flash-attention

Fast and memory-efficient exact attention

flash-linear-attention

Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton

generative-ai-for-beginners

12 Lessons, Get Started Building with Generative AI 🔗 https://microsoft.github.io/generative-ai-for-beginners/

ggml

Tensor library for machine learning

githubdaily

坚持分享 GitHub 上高质量、有趣实用的开源技术教程、开发者工具、编程网站、技术资讯。A list cool, interesting projects of GitHub.

how_to_optimize_in_gpu

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

int8-flash-attention-fmha-quantization

intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

llama.cpp

LLM inference in C/C++

llmsys-paperlist

Large Language Model (LLM) Systems Paper List

migration-falcon

重庆大学计算机学院毕业设计——基于MindSpore的Falcon大模型迁移与性能研究

mindnlp

An opensource NLP library based on MindSpore.

mlc-llm

Enable everyone to develop, optimize and deploy AI models natively on everyone's devices.

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

neural-speed

An innovative library for efficient LLM inference via low-bit quantization

q4m

quantization for mindnlp

riscv-matrix-extension-spec

A matrix extension proposal for AI applications under RISC-V architecture

stable-fast

Best inference performance optimization framework for HuggingFace Diffusers on NVIDIA GPUs.

tensorrt

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

tensorrt-llm

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

transformercompression

For releasing code related to compression methods for transformers, accompanying our publications

transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

triton

Development repository for the Triton language and compiler

zzx