AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

AutoRAG-Optimizer is a from-scratch ML systems project designed to systematically evaluate Retrieval-Augmented Generation (RAG) pipelines under controlled experimental conditions. The framework enables benchmarking of retrieval strategies, chunking configurations, prompt grounding approaches, and computational tradeoffs using real QA datasets.

KBO RAG demo optimized by AutoRAG

recipe RAG demo optimized by AutoRAG

KHU RAG demo optimized by AutoRAG

Automatically optimize prompts for any LLM task. Uses AutoRAG to generate QA ground truth from your documents, then applies PE2 (Ye et al., ACL 2024) which was shown to outperform APE, APO and other automated prompt engineering methods across benchmarks. Works with local models via vLLM or Ollama, or any OpenAI-compatible API.