MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

dev for deepspeed-mii

Experiments with DeepSpeed MII library

Testing DeepSpeed MII

DeepSpeed-MII for UniSpar Project

test

Modificiation of MII

Launch your own high-performance DeepSpeed-MII server for seamless local LLM deployment. This repository provides a Dockerized solution to serve Hugging Face models (e.g., Mistral-7B) with an OpenAI-compatible API, enabling GPU-accelerated, low-latency inference out of the box.

Reproducible benchmarking suite for LLM inference stacks (vLLM, TGI, llama.cpp, Ollama, DeepSpeed-MII) on Kubernetes. Measures throughput, latency, GPU utilisation, and cost-per-token under production-grade K8s conditions including auto-scaling and pod scheduling overhead.