RealVuln: findings, false alarms, coverage and cost

RealVuln tests how well security scanners find known vulnerabilities in a collection of code repositories. It records the bugs found, false alarms, missed bugs and cost. Read each score alongside the repositories tested: a result on the Python subset covers less code than a run across all 140 repositories.

The benchmark uses labelled vulnerabilities

RealVuln is Kolega’s benchmark for security scanners. Version 3.1.0 covers 140 repositories, with 74 TypeScript or JavaScript projects and 66 Python projects. Each repository is tied to a specific commit so the tested code can be identified.

The published dashboard contains 35 scanner entries. Its generation date remains 11 September 2026, and the results were checked again on 8 October. A later review date leaves the original measurement date intact.

Read coverage beside the score

RealVuln’s main score, F3, combines precision and recall while giving recall greater weight. Precision measures how many reported findings match the benchmark’s labels. Recall measures how many labelled vulnerabilities the scanner found.

The dashboard supplies two scoring bases. The micro result covers repositories the scanner completed; the strict result includes the full corpus. Claude Opus 5 scored 67.7 micro F3 on 66 repositories, and 33.1 strict F3 when the full corpus was included. Compare results with similar coverage and keep both figures beside a partial run.

The published results retain their September date

These selected rows come from the dashboard generated on 11 September and checked on 8 October. Costs are US dollars. The model security board places the mapped model results beside other security evaluations.

Model (scanner)	Repos	F3 · strict F3	Precision · recall · $ per 100 lines
Kolega devsec-max v0.1.0 (vendor stack)	140/140	84.4 · 84.4	53.25% · 90.21% · unpublished
GPT Daybreak Blue (gpt-daybreak-blue-codex-cli)	140/140	79.5 · 79.5	63.98% · 81.75% · $0.0739
GPT-5.6 Sol (gpt-5.6-sol-codex-cli)	140/140	74.7 · 74.7	59.33% · 76.92% · $0.0756
Claude Opus 5 (claude-opus-5-cc-agentic-v1)	66/140	67.7 · 33.1	61.95% · 68.40% · $0.0678
Kimi K3 (kimi-k3-agentic-v1)	66/140	59.4 · 28.6	73.02% · 58.20% · $0.0159
GLM-5.3 (glm-5.3-agentic-v1)	136/140	56.9 · 54.6	44.89% · 58.70% · $0.0148
GPT-6 Astra (gpt-6-astra-codex-cli)	140/140	52.1 · 52.1	44.73% · 53.09% · $0.1404
DeepSeek V4.1 Flash (deepseek-v4.1-flash-agentic-v1)	140/140	50.9 · 50.9	52.34% · 50.72% · unpublished
Claude Sonnet 5 (claude-sonnet-5-cc-agentic-v1)	140/140	43.3 · 43.3	52.65% · 42.48% · $0.0176
DeepSeek V4 Flash (deepseek-v4-flash-agentic-v1)	140/140	41.8 · 41.8	56.31% · 40.67% · $0.0005
Gemini 3.5 Flash (gemini-3.5-flash-agentic-v1)	64/140	35.5 · 16.2	89.77% · 33.26% · $0.0270
Gemma 4 31B (gemma4-31b-agentic-v1)	66/140	25.4 · 11.8	90.12% · 23.50% · unpublished
Semgrep (static)	140/140	10.0 · 10.0	11.53% · 9.91% · static

Kolega’s devsec-max entry is a combined vendor system. The other model entries also include the software, prompts and tools used to run them. A score describes that tested system and configuration.

Prompts and run settings differ

The Claude Opus 5 manifest identifies its prompt, repository subset, timeout and cost. Its prompt fingerprint differs from those used for several other rows. Fingerprints identify which prompt was used; compare them before treating two runs as equivalent.

Reasoning settings, completion rates and repeat counts also vary. A run that times out covers less of the requested work. A result from one attempt provides less information about consistency than several repeated attempts under the same conditions.

Use the costs to budget a comparable test

The Opus 5 run cost $90.68 across 66 repositories, about $1.37 per repository. Those are measured costs for that benchmark configuration. The dashboard’s per-line figures are useful for comparing similar runs, while the cost of a new project depends on its code, the tools used and the number of attempts.

Finding a candidate flaw is one part of security work. Researchers still need to reproduce it, establish its impact and check the target’s rules before submitting a report. The bounty-scope guide explains the reward conditions to check alongside the cost.

Questions people ask

What does RealVuln measure?

It measures how accurately a scanner identifies labelled vulnerabilities in 140 repositories. Recall describes the share of labelled bugs found; precision describes the share of reported findings that match the benchmark’s labels. Its headline F3 score gives recall more weight than precision.

Which model leads the published results?

In the dashboard generated on 11 September 2026 and checked on 8 October, GPT Daybreak Blue leads the model rows at F3 79.5 across 140 repositories. Kolega’s combined devsec-max system scores 84.4. Coverage and test settings vary across entries.

How should I read the costs?

Costs describe the particular published runs. Claude Opus 5’s 66-repository run cost $90.68 in total, about $1.37 per repository. A different repository, tool configuration or retry budget can change the bill. Human validation and report preparation are additional work.