Cybench

What the 40 tasks are

Cybench is Stanford’s, from the Center for Research on Foundation Models, first author Andy K. Zhang with Dan Boneh, Daniel E. Ho and Percy Liang among twenty-three authors, published at ICLR 2025 (CONFIRMED from the BibTeX block in the project site’s source). The task set is 40 professional-level capture-the-flag tasks from four competitions, split by category as crypto 16, web 8, reverse engineering 6, forensics 4, misc 4 and pwn 2 (CONFIRMED, same file). Each task ships a description, starter files and an evaluator, and most carry subtasks: intermediate questions with answers, scored in sequence, which turn a binary solve into partial credit.

Three scores per run

Scoring runs in two modes, from the framework README (CONFIRMED, opened 11 September 2026). Unguided mode gives one objective and one binary result; subtask mode reports both the fraction of subtasks solved and a binary score on the final subtask alone. The third published measure is the human difficulty anchor, first solve time: the time the first human team took on that challenge in the original competition, reported as the hardest task an agent solved. A vendor row on the leaderboard carries the end-to-end figure on the tasks the vendor ran; only Cybench’s own runs carry the subtask figures.

The leaderboard on 11 September 2026

Read from data/leaderboard.csv in the site repository, the file the page fetches at load (CONFIRMED, opened). The task count is what the vendor actually ran, which is rarely 40; rows marked card come from a vendor system or model card, HAL from the HAL leaderboard, and the rest are Cybench’s own runs.

Ten older rows follow in the CSV, all run on the full 40 by Cybench itself: GPT-4o 12.5 per cent with subtasks at 28.7, OpenAI o1-preview and Claude 3 Opus 10 per cent (46.8 and 36.8 on subtasks), OpenAI o1-mini 10 per cent as a HAL row adjusted, Llama 3.1 405B, Gemini 1.5 Pro and Mixtral 8x22b 7.5 per cent, Llama 3 70b 5 per cent (CONFIRMED). The top row resolves a figure this site had carried from a card summary: Claude Mythos Preview at 100 per cent, on 35 of the 40 tasks, which is the honest way to state it, saturated on the subset the vendor ran.

A harness leak, caught in public

The two adjusted rows are the most instructive thing on the page, because they are a harness leak caught and corrected in the open. The leaderboard’s own note, verbatim: “The scores for OpenAI o3-mini and OpenAI o1-mini are inflated because HAL likely ran on a fork of the Inspect framework that leaked the answer to a task that both models completed successfully. Their Unguided % Solved scores have been adjusted downward by 2.5% (to 22.5% and 10% respectively) and their FSTs have been updated to reflect their most difficult tasks solved excluding the leaked task (42 min for o3-mini and 11 min for o1-mini)” (CONFIRMED). Every benchmark in this family is built from challenges that were solved and written up before the models were trained, and the leaderboard is the one place the record shows the correction being made rather than described.

The AISI port ships 39 tasks

The UK AI Security Institute’s Inspect Evals carries the most used independent implementation, and it ships 39 tasks, one fewer than the original (CONFIRMED from the port’s README). Its changelog entry of 3 August 2026 records that challenge files now download from pinned upstream commits and are checked against recorded SHA-256 sums, 34 files and about 16.5 MB. The port also warns that the reference Dockerfiles have drifted from the images in use: “These Dockerfiles no longer reproduce the images actually in use. Ten victim images were built from end-of-life bases that stopped resolving (Debian Buster’s removed apt repositories and the delisted openjdk:11-slim tag), so their bases were moved to supported equivalents” (CONFIRMED, verbatim). Indexed search content returned a claim of 90 challenges; the repository, the project page and the AISI port say 40, 40 and 39, and the repository wins (CONTESTED, resolved).

What a Cybench percentage needs beside it

So a Cybench percentage carries three facts at once: the task count, the harness, and whether the challenges ran as the exact pinned versions, and two runs quoting Cybench can differ on all three. The Model Security Capability board prints the task count beside every Cybench row for that reason, and marks each card row as the vendor’s own run. The port’s verification route is the other half of the answer: run the solution variants through inspect_cyber/verify_solutions to confirm a challenge is still solvable before trusting a score on it.

Where it sits beside BountyBench and RealVuln

The same Stanford lineage produced BountyBench, announced on the Cybench site as “an all-new, real-world BountyBench that evaluates offensive and defensive cybersecurity agents on vulnerability detection, exploitation, and patching with dollar impact” (CONFIRMED, verbatim). Each task is a real, paid bug bounty imported with its award; the task repository’s .gitmodules lists 31 codebases, six more than the 25 systems the paper describes, and three metadata files read in full carry awards of $1,080, $750 and $540 for disclosure with patch awards at a quarter of each (CONFIRMED). Its published leaderboard, SINGLE because the site refused this desk, has the same agents patching at around 90 per cent and detecting at 12.5 per cent, with every model row from 2025, so nothing on it measures the 2026 generation. Detect is the task closest to bug hunting and it is where every agent measured scores lowest.

The two families measure different halves of one job. Cybench, CVE-Bench, CyberGym and ExploitBench measure whether one bug can be driven to a working proof, the step a bounty programme pays for. RealVuln measures triage quality across a large corpus: how many real defects the agent names, how much chaff arrives with them, at what cost per repository. BountyBench spans both, and its own numbers say the two skills come apart. The shared methodology point is the stronger one: RealVuln publishes a content-hashed prompt, a pinned corpus and a per-model cost; ExploitBench publishes locked decisions, seed counts and eleven audit checks; the Cybench leaderboard prints the task count and corrects a leak in public; most vendor charts publish a percentage.