🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

LLM inference in C/C++

A high-throughput and memory-efficient inference and serving engine for LLMs

Inference code for Llama models

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Port of OpenAI's Whisper model in C/C++

TypeScript-first schema validation with static type inference

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

Official inference framework for 1-bit LLMs

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Cross-platform, customizable ML solutions for live and streaming media.

SGLang is a high-performance serving framework for large language models and multimodal models.

Official inference repo for FLUX.1 models

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Faster Whisper transcription with CTranslate2

Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Inference Llama 2 in one file of pure C

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Run OpenClaw more securely inside NVIDIA OpenShell with managed inference

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

A list of free LLM inference resources accessible via API.

High-performance In-browser LLM Inference Engine

Machine Learning Engineering Open Book

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Inference code for CodeLlama models

AirLLM 70B inference with single 4GB GPU

🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.