Choosing a SAST tool without being fooled

Two tools scan your application. One returns eight alerts, the other sixteen. Which should you choose?
Without knowing what they analyzed, what they missed and how long their results take to handle, those counts cannot settle the decision. Here is how to build a trial that can.

This article builds on SAST in depth, but stands on its own.

Start with the work to be done

The right tool must cover your code and fit your team. Before a demo, write down three constraints:

• Required code coverage. Languages, frameworks, generated code, repository size and whether you can build it. A “Java supported” checkbox does not tell you whether your web inputs and internal libraries are understood.
• The decision point. A check before merging and a deep overnight analysis have different time budgets. Specify what should block a release and who can resolve a disputed alert.
• Operating conditions. Whether code may leave your network, Internet access, required permissions, budget and who will maintain the rules.

An engine detects. A platform also provides result management, permissions and integrations. For each offer, distinguish what is included, what needs another product and what your team would have to build.

The perfect-score trap

One public benchmarkbenchmark: a shared set of tests used to compare tools, or reference test suite, is the OWASPOWASP: a foundation that publishes free application security guides and tools Benchmark Java 1.2, published by OWASP (Open Worldwide Application Security Project), a non-profit foundation.

The counts above come from the expected results file for version 1.2.

Its score is the share of flaws found minus the share of safe cases wrongly flagged: a tool that flags everything scores zero.
That is a good idea.

But this Java corpus is synthetic (there is also a Python one): an excellent score says little about how the tool behaves on your code. And beware of Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Tuning for this corpus can improve the score without improving detection on your application.

Same limit for NIST's Juliet suite, tens of thousands of synthetic cases in C, C++ and Java: useful for testing rules, insufficient on its own for choosing a tool.

SAST engines, one minute each

With the vocabulary from the deep dive (syntax tree, taint, sensitivities), these engines are easier to read (profiles checked on 8 October 2026, sources linked from tool names). They are not interchangeable: first select those covering your languages and defect types. For each one: what it builds, what it looks for, and its limitation.

For Checkmarx, Snyk Code and SonarQube, verify the edition, languages and models included in the vendor's dated offer.

Judging a SAST tool on your code

On a copy of your repository, fix the scope, tool versions, rules, exclusions and available resources. Keep commands and reports so you can repeat the trial. Compare defaults first, then record the time spent tuning them.

Prepare a trial you can repeat

1. Choose the scope. A representative repository, its dependencies and build commands. Keep the same copy for every candidate.
2. Establish reference cases. Known vulnerabilities, their fixed versions and safe code that resembles a flaw. Include your actual framework. Record expected results before running the tools.
3. Run two passes. First without corpus-specific tuning, then with the same configuration time budget. Hold a few cases back to check that models work beyond the examples already shown.
4. Have your team triage. A developer and an AppSec analyst review the same alerts without the vendor's conclusion in front of them. Record disagreements, time to understand and missing evidence.
5. Keep the evidence. Versions, configuration, skipped files, errors, results and decisions. An alert disappearing after a failure or exclusion is not a fix.
6. Detail of the five tests in the table. Flows: check each alert and its disappearance after the fix, with identical configuration. Sample: 50 alerts, or all if fewer, measuring known flaws found and missed separately. Sanitizer: the unprotected case and the one with unsuitable protection must still trigger. Unreadable file: check the diagnostic, the partial or absent analysis, and the exclusion reason. Pull request: separate first and cached analyses, and record total time, memory and any timeouts.

Five questions, the evidence to demand, and the test to run yourself:

Read two reports without misreading the results

Fictional example, unrelated to the tools listed. The reference set contains 12 vulnerable cases and 12 cases without the flaw being tested. Both engines analyze every file. Count one alert per case after grouping duplicates.

A provides fewer alerts to review but misses four flaws. In this example, B finds all eight flaws found by A and three others. It needs eight additional reviews: three concern extra real flaws, five concern false alerts. The decision also depends on which flaws were missed and your capacity to handle the results.

These percentages do not carry over to your repository, where unknown flaws remain unknown: recall is measured on your reference cases. Keep unresolved alerts separate.

The trap to avoid:zero alerts with zero files analyzed is not a clean result. And 100% precision on eight alerts does not mean every flaw was found.

Test the work after the alert

Take a real alert from the pilot and give it to whoever would fix it. Can they locate the input, the dangerous operation and the steps between them, explain the fix and verify the corrected behavior beyond the signal simply disappearing? Then follow the same defect through changed line numbers, a new scan and a fix: its history must stay attached to the right issue. Also try a justified exception: who approves it, when does it expire and what happens if the code changes?

If the vendor offers an AI-generated fix

Evaluate the fix as a code change: it must remove the cause, preserve expected behavior and pass tests. An ignore comment or a disabled rule can make an alert disappear without fixing the problem.

Ask what code and context are sent to the model, where they are retained and whether this feature can be disabled. Measure the time spent reviewing the fix, not just generating it.

Finally, simulate an interrupted or incomplete analysis in the pilot. The release check must follow your chosen policy and explain the failure. A green check without usable results distorts the entire comparison.

Before you sign

Set your criteria before the trial: required languages, known flaws to detect, triage budget and acceptable time per change. A good overall score cannot compensate for a required language left unanalyzed. Test a SARIF export in the tool that will receive your results: locations, traces and identifiers must remain usable. Importers do not all support the same fields, as illustrated by the GitHub documentation. Separately check the recovery of decisions, comments and assignees. Finally, ask for the total cost (license, execution, tuning, triage), the data sent outside your environment and its retention period. Record gaps against your criteria before deciding.

Wrapping up

An alert count does not settle a choice between two SAST tools. What matters: what the tool actually read, the known flaws it finds in your code, the time it takes to decide each alert, and what it does when it fails. Set these criteria before the demo, replay the trial on your repository, and keep the proof.

Words in this article · 4 words

Benchmark

A shared set of tests used to compare tools.

Precision

The share of a tool's alerts that are real flaws.

Recall

The share of real flaws a tool finds.

SARIF

A standard format for exchanging code analysis results between tools.

To revisit the basics: SAST explained simply. And for imported blocks: Choosing an SCA tool without being fooled.

Start here: SAST in depth: engines, taint and limits