A modular framework for benchmarking LLMs and agentic strategies on security challenges across HackTheBox, TryHackMe, PortSwigger Labs, Cybench, picoCTF and more.

WMDP is a LLM proxy benchmark for hazardous knowledge in bio, cyber, and chemical security. We also release code for RMU, an unlearning method which reduces LLM performance on WMDP while retaining general capabilities.

SecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).

LLM agent solving traces, leaderboards, and benchmark results across security CTF and hacking platforms

Automated Benchmarking of LLM Agents on Real-World Software Security Tasks [NeurIPS 2025]

Mithra Scanner is an interactive API testing tool for prompt injection, refusal detection, and LLM security benchmarking. It supports YAML-based rule definitions, custom refusal lists, REST API integration, and provides detailed CLI output for security testing of language model endpoints.

Red Team AI Benchmark: Evaluating Uncensored LLMs for Offensive Security

A robust framework to benchmark LLMs specifically for penetration testing use-cases, offering a significant leap forward in the security assessment process.

Phare is a LLM benchmark that evaluates models across key AI security & safety dimensions

Kidnapp-AI-Benchmark is a modular, extensible framework designed to systematically test and evaluate privacy leakage, data extraction, and adversarial vulnerabilities in large language models (LLMs) and other generative AI systems. Built for red teamers, penetration testers, and AI security researchers.

Benchmarking 12 LLMs for vulnerability research

A multi-LLM benchmark suite for evaluating security analysis and vulnerability detection capabilities across OpenAI, Anthropic, Google's models.

A comprehensive benchmark system for evaluating whether Large Language Models (LLMs) can be tricked into ignoring security vulnerabilities through deceptive code patterns and misleading comments.

A repository dedicated to benchmarking lightweight, open-source large language models (LLMs) for their effectiveness in providing security guidance. This project uses the SECURE dataset as a foundation to replicate research and evaluate selected models on predefined cybersecurity tasks.

A benchmark for cyber security knowledge evaluation on LLM

AI SysAdmin Trust Benchmark - Comprehensive testing suite for evaluating LLM competence in system administration. Real-world scenarios covering setup, security, networking, monitoring, and troubleshooting.

TrustMH_Bench is a trustworthiness benchmark for general-purpose and mental-health LLMs in mental health settings. It evaluates models across fairness, privacy, reliability, security, crisis identification and escalation, ethics, robustness, and sycophancy. Supports standardized, reproducible evaluation for researchers and developers.

This repository presents the results of a comprehensive multi-LLM security benchmark study evaluating the effectiveness of the AEGIS PALADIN 6-Layer Defense System as a deterministic guardrail across six major Large Language Models.

benchmarks the security and performance of open-source large language models (LLMs) from Hugging Face

AART provides security researchers, AI labs, and red teams with a structured framework for conducting thorough adversarial evaluations of LLM systems. The framework implements a multi-dimensional assessment methodology that systematically probes model boundaries, quantifies security vulnerabilities and benchmarks defensive robustness in frontier AI

A curated list of papers on privacy, security, and compliance in LLM-based agent systems — attacks, defenses, benchmarks, and regulatory frameworks.

Security benchmarking of low param (< 3B) LLMS using Nvidia's garak tool.

TOSSS, an extensible LLM security benchmark based on the CVE database

This repository explores the security vulnerabilities of large language models (LLMs) to prompt injection attacks. It includes a research paper, benchmarks, attack/defense taxonomies, and illustrations of both direct and indirect prompt injections. Ideal for researchers, developers, and security practitioners working on LLM safety.

α³-SecBench is a large-scale benchmark for evaluating security, resilience, and trust of LLM-based UAV agents under realistic adversarial conditions in 6G-enabled networks, featuring layered attack taxonomies and CWE-aligned evaluation.

No description available

Benchmarking LLM Security tools

Benchmarking the security of various LLMs

Financial Security Knowledge Understanding Benchmark for LLMs

LLM Security Benchmark Hub - 6개 벤치마크 모니터링 + 화이트해커 리소스 90개