Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

An MCP server exposing full Chrome DevTools Protocol debugging: breakpoints, step/run, call stacks, eval, and source maps.

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

Claude Code plugin: 115 commands, 25 agents, 64 scripts, 67 MCP servers, 143 reference files. Eval/QA layer (hallucination detection, claim verification, A+ through F grading). Multilingual (Sarvam AI, DeepL, Google Cloud Translation). Full execution with approval workflow.

A complete, driver based RAG pipeline for Laravel with pgvector & sqlite-vec, streaming, agentic retrieval, hybrid search, evals, MCP server, Filament admin.

Local-first agent runtime for MCP workflows with explicit trust controls, replayable runs, and built-in evals.

Persistent memory for AI agents. 107 MCP tools for remember, recall, rollback, audit, knowledge graph, eval, cortex, replay, governed actions, threat intel, auto-defense, Runtime v2 agents/missions, and more. Works locally (zero config) or with Novyx Cloud.

An open-ended eval framework for mcp.run tools

Lightweight eval framework for MCP servers, built on mcp-agent

Lightweight framework for generating, running, and reviewing MCP evals.

EVE Online Market MCP Server - A Model Context Protocol server for accessing EVE Online market data through ESI API

The agent eval standard for MCP — score output quality, catch safety failures, enforce cost budgets

Playwright-based testing and eval framework for MCP servers with LLM-as-a-judge

A local MCP server for accessing the EVE Online ESI API

Evals using Braintrust for Stripe MCP servers

A Go library and CLI for evaluating Model Context Protocol (MCP) servers using Claude.

Bridging Generative AI and Classical ML. A production-grade multi-agent system using ReAct orchestration, MCP, A2A, and LMMs for hyper-local property valuation with Evals.

A step-by-step educational repo that teaches AI agent development from scratch—agent loop, MCP, reflection, tool use, evals, and multi-agent—with concrete examples and code.

Starter kit for MCP-powered AI agents: Includes sample client, eval tools, and easy integration guide for Claude Code & beyond.

🚀 The Definitive Field Manual for AI Engineering. 12 chapters covering RAG, Agents, MCP, Evals, and Token Economics. 50+ proven patterns for shipping mission-critical LLM products. Vendor-agnostic, senior-level documentation for the 2026 AI tech stack.

MCP Evals

A CLI tool for evaluating MCP servers

MCP Evals CLI (Deno + Cliffy): import traces, browse viewer, ask, config

Order processing API for Drift MCP evals

MCP Server for EVE fleet manager

MCP server for EVE Online market data

Code for the blog post "AI Eval for MCP in AIOps".

This should be the first complete agent in a harness w/ MCPs, RAG, Session File Memory, Skills, Usage Meter, Evals etc. FIRST AGENT: GEMINI ARCHITECT

A robust MCP server for Elixir/BEAM applications — logs, eval, docs over persistent TCP with auto-reconnect

Adversarial eval harness for any LLM agent pipeline — Claude, OpenAI, or your own. CLI + REST API + MCP server for Cursor/Antigravity.