Manual CVE_R is the lead signal

SAST Benchmark Leaderboard

We cross-reference the 165-case Java CVE Benchmark with OWASP Benchmark F1 results. Manual Check (S_M-C) detection rates stay front and center, while Approx. FP and the gap between vendor claims and measured detections remain visible at a glance.

Manual Best

Horusec

12.7% CVE_R (Manual)

Avg. Manual CVE_R

6.2%

Across all 7 tools / 165 CVEs

Manual coverage

12.7

Manual findings / 165 CVEs

Filters

Switch between license, freshness, and indicator priorities.

Supported languages: Java (multi-language UI planned)

License

Indicator

Last updated

Languages

CVE_R x OWASP F1 bubble chart

X: Manual CVE_R% / Y: OWASP F1% / bubble size: speed tier (fast < mid < slow).

fast: insider, contrastslow: semgrep, codeql

Leaderboard (manual-first sort)

Sorted by Manual CVE_R counts, with a helper score (Manual 60% + F1 30% + inverse FP 10%). High Approx. FP or over-claim tools surface with badges so non-experts can spot risk quickly.

Horusec

OSS

Manual 12.7%OWASP F1 49.0%Approx.FP 28.6%Over-claim 80.4%

Manual CVE_R (S_M-C after review)

12.7%21 / 165 CVEs

Composite score

29.5

Updated: 2023-08-15Roughly 5-15 min depending on project size

Scenario	Count	%
S_F-A	57	34.5%
S_F-C	38	23.0%
S_M-A	46	27.9%
S_M-C	28	17.0%
Manual (S_M-C verified)	21	12.7%

Speed: mid / Roughly 5-15 min depending on project size

View details

SpotBugs+FindSecurityBugs

OSS

Manual 10.3%OWASP F1 82.8%Approx.FP 94.1%Over-claim 86.7%

Manual CVE_R (S_M-C after review)

10.3%17 / 165 CVEs

Composite score

31.6

Updated: 2023-07-30Roughly 5-15 min depending on project size

Scenario	Count	%
S_F-A	34	20.6%
S_F-C	26	15.8%
S_M-A	31	18.8%
S_M-C	19	11.5%
Manual (S_M-C verified)	17	10.3%

Speed: mid / Roughly 5-15 min depending on project size

View details

CodeQL

OSS

Manual 6.7%OWASP F1 79.8%Approx.FP 45.5%Over-claim 92.1%

Manual CVE_R (S_M-C after review)

6.7%11 / 165 CVEs

Composite score

33.4

Updated: 2023-10-01Long runs (Semgrep 230-274s, CodeQL queries may reach 24h)

Scenario	Count	%
S_F-A	24	14.5%
S_F-C	15	9.1%
S_M-A	20	12.1%
S_M-C	13	7.9%
Manual (S_M-C verified)	11	6.7%

Speed: slow / Long runs (Semgrep 230-274s, CodeQL queries may reach 24h)

View details

Semgrep

OSS

Manual 5.5%OWASP F1 29.7%Approx.FP 44.4%Over-claim 88.6%

Manual CVE_R (S_M-C after review)

5.5%9 / 165 CVEs

Composite score

17.8

Updated: 2023-12-01Long runs (Semgrep 230-274s, CodeQL queries may reach 24h)

Scenario	Count	%
S_F-A	60	36.4%
S_F-C	31	18.8%
S_M-A	37	22.4%
S_M-C	17	10.3%
Manual (S_M-C verified)	9	5.5%

Speed: slow / Long runs (Semgrep 230-274s, CodeQL queries may reach 24h)

View details

SonarQube (CE)

OSS

Manual 5.5%OWASP F1 27.0%Approx.FP 55.6%Over-claim 90.8%

Manual CVE_R (S_M-C after review)

5.5%9 / 165 CVEs

Composite score

15.8

Updated: 2023-09-05Roughly 5-15 min depending on project size

Scenario	Count	%
S_F-A	22	13.3%
S_F-C	12	7.3%
S_M-A	16	9.7%
S_M-C	10	6.1%
Manual (S_M-C verified)	9	5.5%

Speed: mid / Roughly 5-15 min depending on project size

View details

Insider

Commercial

Manual 2.4%OWASP F1 13.9%Approx.FP 100.0%Over-claim 95.8%

Manual CVE_R (S_M-C after review)

2.4%4 / 165 CVEs

Composite score

5.6

Updated: 2023-09-20Scan completes ~<=5 min (LoC < 50k)

Scenario	Count	%
S_F-A	29	17.6%
S_F-C	11	6.7%
S_M-A	19	11.5%
S_M-C	9	5.5%
Manual (S_M-C verified)	4	2.4%

Speed: fast / Scan completes ~<=5 min (LoC < 50k)

View details

Contrast Codesec Scan

Commercial

Manual 0.6%OWASP F1 84.4%Approx.FP 100.0%Over-claim 99.1%

Manual CVE_R (S_M-C after review)

0.6%1 / 165 CVEs

Composite score

25.7

Updated: 2023-11-10Scan completes ~<=5 min (LoC < 50k)

Scenario	Count	%
S_F-A	3	1.8%
S_F-C	2	1.2%
S_M-A	2	1.2%
S_M-C	1	0.6%
Manual (S_M-C verified)	1	0.6%

Speed: fast / Scan completes ~<=5 min (LoC < 50k)

View details