A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

The papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey.

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

Run evaluation on LLMs using human-eval benchmark

A curated list of Human Preference Datasets for LLM fine-tuning, RLHF, and eval.

Claw-Eval is an evaluation harness for evaluating LLM as agents. All tasks verified by humans.

LLM fine-tuning and eval

Sample notebooks and prompts for LLM evaluation

No description available

A Survey on Benchmarks of Multimodal Large Language Models

Scenario-based Evaluation dataset for LLM (beta)

This project benchmarks 41 open-source large language models across 19 evaluation tasks using the lm-evaluation-harness library.

Serbian LLM Eval.

Top papers related to LLM-based agent evaluation

大语言模型评估平台，支持多种评估基准、自定义数据集和性能测试。支持基于自定义数据集的RAG评估。

Simple LLM Evaluation Using LLM As a Judge 👩‍⚖️

llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection

No description available

[ACL'24] Code and data of paper "When is Tree Search Useful for LLM Planning? It Depends on the Discriminator"

Fine-tunes a student LLM using teacher feedback for improved reasoning and answer quality. Implements GRPO with teacher-provided evaluations.

Large-language Model Evaluation framework with Elo Leaderboard and A-B testing

Open Source Embeddings Optimisation and Eval Framework for RAG/LLM Applications. Documentations at https://docs.vectorboard.ai/introduction

No description available

No description available

Performs benchmarking on two Korean datasets with minimal time and effort.

Benchmarking Large Language Models for FHIR

Code space for 'Evaluation of large language models for discovery of gene set function'

Using LLM to evaluate MMLU dataset.

A lightweight framework for evaluating visual-language models.