mcp-evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

aievalsmcp

MIT License

Created on Apr 23, 2025

Updated on Mar 30, 2026

Stars

127

Forks

13

Watchers

127

Open Issues

6

Repository Health Score

💛

70/100

Good

Overall repository health assessment

Score Breakdown

Activity

Active development - updated this week

30/30

100%

Issues Analytics

8

Total Issues

All time

6

Open

75% of total

2

Closed

Recent Commits

Merge pull request #8 from mclenhard/feat/add-e2e-tests

mat lenhard•9 months ago

9737963View on GitHub

[feat] add tests

Matthew Lenhard•9 months ago

3c435beView on GitHub

Merge pull request #7 from mclenhard/feat/add-yaml-based-tests

mat lenhard•9 months ago

65310f4View on GitHub

[feat] add yaml based evals

Matthew Lenhard•9 months ago

97a1f16View on GitHub

[chore] remove latest tag from npm

Matthew Lenhard•10 months ago

614087dView on GitHub

[chore] remove table parsing

Matthew Lenhard•10 months ago

23ad1c4View on GitHub

[feat] bugfix fix npm package

Matthew Lenhard•10 months ago

5b75951View on GitHub

[feat] bugfix fix npm package

Matthew Lenhard•10 months ago

88cae03View on GitHub

[feat] format action comment as table instead of json

Matthew Lenhard•10 months ago

09435f7View on GitHub

Merge pull request #4 from mclenhard/feat/add-tracing

mat lenhard•10 months ago

fd27959View on GitHub

[feat] add metrics

Matthew Lenhard•10 months ago

5f0286dView on GitHub

[feat] add metrics exporter

Matthew Lenhard•11 months ago

8a25156View on GitHub

[feat] intilize metrics

Matthew Lenhard•11 months ago

f5bd112View on GitHub

Merge pull request #3 from SecretiveShell/add-free-openai

mat lenhard•11 months ago

7003942View on GitHub

Update README.md

TerminalMan•11 months ago

0b366b3View on GitHub

View all commits

Community

127 stars, 13 forks

0/30

0%

Documentation

Has description, wiki, license

20/20

100%

Maintenance

4.7% issue ratio

20/20

100%

Health score is calculated based on activity, community engagement, documentation quality, and maintenance practices

Languages

TypeScript

79.7%

JavaScript

20.3%

Dependencies32 total

20

Production

11

Development

Production Dependencies (20)

@actions/core^1.10.0

@ai-sdk/anthropic^1.2.12

@ai-sdk/openai^1.3.17

@anthropic-ai/sdk^0.8.0

Top Contributors

1

mclenhard

User

29

commits

2

SecretiveShell

User

1

commits

Languages

TypeScript

79.7%

JavaScript

20.3%

Issues Analytics

8

Total Issues

All time

6

Open

75% of total

2

Closed

Dependencies32 total

20

Production

11

Development

Production Dependencies (20)

@actions/core^1.10.0

@ai-sdk/anthropic^1.2.12

@ai-sdk/openai^1.3.17

@anthropic-ai/sdk^0.8.0

@modelcontextprotocol/sdk^1.11.0

@opentelemetry/api^1.9.0

@opentelemetry/exporter-trace-otlp-http^0.200.0

@opentelemetry/resources^2.0.0

@opentelemetry/sdk-node^0.200.0

@opentelemetry/sdk-trace-base^2.0.0

@opentelemetry/semantic-conventions^1.33.0

@types/js-yaml^4.0.9

ai^4.3.9

chalk^4.1.2

dotenv^16.3.1

express^5.1.0

js-yaml^4.1.0

openai^4.24.1

prom-client^15.1.3

tsx^4.19.3

Development Dependencies (11)

@types/express^5.0.1

@types/jest^30.0.0

@types/node^20.10.0

@typescript-eslint/eslint-plugin^6.13.0

@typescript-eslint/parser^6.13.0

eslint^8.54.0

jest^29.7.0

react^19.1.0

ts-jest^29.1.1

ts-node^10.9.1

typescript^5.3.2

Peer Dependencies (1)

react^19.1.0

mcp-evalsv1.0.18

25% of total

1h

Avg Close Time

Fast response ✅

Issues Activity: Last 6 months

Hottest Issues

1

#1 Add OLLAMA Support

4

open

2

#6 Pass in custom grading prompt

2

open

3

#9 Feature request: configure system prompt

1

closed

4

#5 Extend documentation with screenshots

1

open

5

#12 Integration proposal: AgentTrust quality verification

0

open

Top Contributors

1

mclenhard

User

29

commits

2

SecretiveShell

User

1

commits

Recent Commits

Merge pull request #8 from mclenhard/feat/add-e2e-tests

mat lenhard•9 months ago

9737963View on GitHub

[feat] add tests

Matthew Lenhard•9 months ago

3c435beView on GitHub

Merge pull request #7 from mclenhard/feat/add-yaml-based-tests

mat lenhard•9 months ago

65310f4View on GitHub

[feat] add yaml based evals

Matthew Lenhard•9 months ago

97a1f16View on GitHub

[chore] remove latest tag from npm

Matthew Lenhard•10 months ago

614087dView on GitHub

[chore] remove table parsing

Matthew Lenhard•10 months ago

23ad1c4View on GitHub

[feat] bugfix fix npm package

Matthew Lenhard•10 months ago

5b75951View on GitHub

[feat] bugfix fix npm package

Matthew Lenhard•10 months ago

88cae03View on GitHub

[feat] format action comment as table instead of json

Matthew Lenhard•10 months ago

09435f7View on GitHub

Merge pull request #4 from mclenhard/feat/add-tracing

mat lenhard•10 months ago

fd27959View on GitHub

[feat] add metrics

Matthew Lenhard•10 months ago

5f0286dView on GitHub

[feat] add metrics exporter

Matthew Lenhard•11 months ago

8a25156View on GitHub

[feat] intilize metrics

Matthew Lenhard•11 months ago

f5bd112View on GitHub

Merge pull request #3 from SecretiveShell/add-free-openai

mat lenhard•11 months ago

7003942View on GitHub

Update README.md

TerminalMan•11 months ago

0b366b3View on GitHub

View all commits

GitHub Explorer

mcp-evals

Score Breakdown

Production Dependencies (20)

Production Dependencies (20)

Development Dependencies (11)

Peer Dependencies (1)

Issues Activity: Last 6 months

Hottest Issues