Cybench | YFarmX

Cybench tests whether an AI agent can solve security challenges by finding and using a vulnerability. The original benchmark contains 40 tasks. Published scores use different subsets and test setups, so read the number of tasks attempted alongside the percentage solved.

Cybench contains 40 security challenges

Cybench tests AI agents on professional capture-the-flag tasks. In these exercises, an agent must work through a security problem and produce the required answer or exploit result. The original set contains 40 tasks across cryptography, web security, reverse engineering, forensics and other categories.

The benchmark records complete solves and progress through subtasks. Its project page distinguishes an agent completing the whole challenge from one answering intermediate questions with extra guidance.

Compare scores on the same task set

The leaderboard, checked on 8 October 2026, retains the following selected results. A system-card entry is a result reported by the model’s vendor; HAL and Cybench entries come from the named evaluation projects.

Model	Tasks run	End-to-end solved	Row source
Claude Mythos Preview	35	100%	card
Claude Opus 4.7	35	96%	card
Claude Opus 4.6	37	93%	card
Claude Opus 4.5	39	82%	card
Muse Spark	40	65.4%	card
Claude Sonnet 4.5	39	60%	card
Grok 4	40	43%	card
Claude Opus 4.1	39	42%	card
Grok 4.1 Thinking	40	39%	card
Claude Opus 4	37	38%	card
Claude Sonnet 4	37	35%	card
Grok 4 Fast	40	30%	card
OpenAI o3-mini	40	22.5%	HAL, adjusted
Claude 3.7 Sonnet	40	20%	HAL
GPT-4.5-preview	40	17.5%	HAL
Claude 3.5 Sonnet	40	17.5%	Cybench, subtasks 43.9%

Claude Mythos Preview’s 100% result covers 35 tasks. Claude Opus 4.5’s 82% covers 39. Those task counts establish different coverage; judging relative difficulty also requires the identities of the tasks and the test conditions. Keep the model, task set and run configuration together when comparing scores.

The software running the test can change the result

The Cybench project records an answer leak affecting two HAL results. It reduced OpenAI o3-mini and o1-mini by 2.5 percentage points, to 22.5% and 10%, after excluding the leaked task. The project’s note attributes the problem to the evaluation setup.

The UK AI Security Institute’s Inspect Evals implementation contains 39 tasks and documents pinned challenge downloads and changes to older container images. Challenge versions, available tools and time limits belong beside the result because they shape what the agent can do.

A solved challenge measures one part of security work

Cybench measures performance on prepared security challenges. RealVuln measures vulnerability detection and false alarms across code repositories. BountyBench uses tasks based on real vulnerabilities to evaluate detection, exploitation and patching.

These tests cover different stages of research. A bounty submission also needs an eligible target, a reproducible impact and compliance with the programme’s policy. Use the model security board to compare the available evidence for the particular work you need to do.

Questions people ask

What is Cybench?

Cybench is a benchmark of 40 professional-level capture-the-flag security tasks developed by Stanford researchers. It evaluates complete task solves and progress through intermediate subtasks.

What did Claude Mythos Preview score?

The leaderboard checked on 8 October 2026 lists Claude Mythos Preview at 100% end-to-end solved across the 35 tasks in its vendor-reported run. The original benchmark contains 40 tasks.

Why do task counts differ?

Published evaluations select different subsets or use different implementations. The original Cybench has 40 tasks; the UK AI Security Institute’s Inspect Evals implementation lists 39. The tested task set belongs beside every score.