SAST in depth: engines, taint and limits

In the introduction, we stayed at the counter: an over-obedient barista, a drop of ink, a game.
Here we go into the kitchen, and it is going to smell of engine oil.

Run three SAST tools on the same repositoryrepository: where a project's code lives, with its history, for example on GitHub: you will get three different lists. That is the subject of this article: what each engine builds from the code, what it follows, what slips past it, and why none will ever be perfect. It builds on SAST explained simply, but stands on its own.

SELECT * FROM invoices WHERE customer = 'MARTIN'

The example application

Here are three files from an imaginary web application, written in Python with Flask, a common tool for building websites. They contain four real vulnerabilities and two traps: code that looks like an injection but is safe.

Before looking at what the tools see, act as the analyzer: it is up to you to say which of the numbered lines are real flaws. A good analyzer flags the former and stays quiet on the latter.

def hash_password(password):
L1    return hashlib.md5(password.encode()).hexdigest()

@app.get("/order")
def order():
    number = request.args.get("id")
L2    sql = "SELECT * FROM orders WHERE id = " + number
    return db.execute(sql)

@app.get("/search")
def search():
    return find_products(request.args.get("q"))

@app.get("/export")
def export():
    name = html.escape(request.args.get("f"))
L3    return open("/srv/exports/" + name).read()

@app.get("/products")
def products():
    sort = request.args.get("sort")
    if sort not in ("name", "price"):
        abort(400)
L4    return db.execute(f"SELECT * FROM products ORDER BY {sort}")

TABLE = "products"

def find_products(term):
L5    return db.execute(f"SELECT * FROM {TABLE} WHERE name LIKE '%{term}%'")

def count():
L6    return db.execute(f"SELECT COUNT(*) FROM {TABLE}")

Compare configurations with the buttons above the code. The table summarizes their results on these six lines:

A result limited to this example.These five fictional configurations isolate analysis choices (a real tool combines them differently), and configuration 5's four out of four with no false alerts applies only to these six lines. For example, replace L2's Flask input with portal.field("client"), a fictional internal API unknown to the taint model. Without a model linking that call to external input, this analysis may miss the injection: tracking across files and recognizing validation checks are not enough.

Two words to judge a tool

A tool can shine on one and fail on the other: flagging everything gives perfect recall and very low precision, flagging nothing never raises a false alarm but finds zero flaws. The OWASP Benchmark score combines the two: true positive rate − false positive rate. Flagging everything gives 100% − 100% = 0.

Configuration 1: patterns find everything, and a bit too much

4 flaws of 42 false alerts

A classic set of pattern rules looks for shapes: "an SQL query made by pasting pieces of text together", "a file opened with a path made the same way", "a call to MD5". On our application, it finds all four vulnerabilities, but also flags L4 and L6, which have exactly the shape of an injection without being one. It does not know where values come from.

Configuration 2: taint analysis cuts the noise, and misses flaws

2 flaws of 41 false alert

Taint analysis adds the missing question: where does the value come from? TABLE is a constant, so L6 disappears. L2 is still found, because the input and the dangerous output are in the same function, and so is L1 (MD5): it is a pattern rule, which taint analysis does not replace.

But two vulnerabilities drop out. L5, because the input is in routes.py and the dangerous output in repository.py: a tool that analyzes function by function does not connect the two. And L3, because the value goes through html.escape, which a tool that does not tell contexts apart takes for a sanitizer.

That is the trap of this configuration: noise drops (1 false alarm instead of 2), but precision stays the same (2 correct alerts out of 3, against 4 out of 6) and recall falls. Fewer alerts does not mean better alerts. To place the well-known tools, see Choosing a SAST tool without being fooled.

Configuration 3: following data from one file to another

3 flaws of 41 false alert

To find L5 again, the tool must connect two files:

It needs a call graphgraph: a diagram of points linked by arrows; here, who calls whom in the program, the map of "who calls whom" in the program, and function summaries: one card per function saying "my first parameter reaches an SQL query". The summary is computed once, then reused at every call, which is what makes analysis scale.

The hard part is the map itself. Here, the call is direct; in a real application, it often goes through indirect mechanisms, where the function actually called is only known at runtime. How each tool guesses "who calls whom" explains a large share of the differences between tools.

Configuration 4: a sanitizer is only valid for its context

4 flaws of 41 false alert

A protection is valid only for its intended uses and assumptions:

../../etc/passwd climbs up the server's folders to a sensitive file, and html.escape lets it through. To find L3 again, the tool must tie each sanitizer to the type of operation it protects.

Remember:a tool that misses L3 reports nothing, and nothing looks more like safe code than a missed vulnerability. It is one of the clearest gaps between tools, and one of the hardest to see in a demo.

Configuration 5: understanding the developer's checks

4 flaws of 40 false alerts

L4 is safe: the value of sort is checked against a list of allowed values before reaching the query. To stay silent, the tool must understand that the abort(400) line, which sends an error back to the visitor, cuts the path, and that the value reaching the query can only be name or price.

This configuration recognizes the check in our example. Real checks take many forms: checks against a text pattern, conversion to a number, functions written by the team. An honest tool lets you declare your own sanitizers and mark an alert as false, while keeping a written record of the decision.

Under the hood: from text to graph

A SAST tool does not read code the way you read this article. It first turns it into a series of representations, each able to answer a different question. Take five lines:

def search():
    q = request.args.get("q")
    if not q:
        q = "coffee"
    return db.execute("SELECT * FROM products WHERE name = '" + q + "'")

1. AST

   The syntax tree is the grammar: db.execute(…) is a call, whose argument is a concatenation of three pieces. Configuration 1 pattern rules work here, without knowing where q comes from.

2. CFG

   The control flow graph is the possible order of execution: after the if, two paths split and join again. Either q comes from the address, or it is “coffee”.

3. SSA

   SSA form (Static Single Assignment) gives every assignment a fresh name: q₁ = request.args.get("q"), q₂ = "coffee", and where the paths join, q₃ = φ(q₁, q₂), “one or the other depending on the path taken”. A φ that mixes ink and clear water gives ink: alert.

4. Calls

   The call graph links functions together, from one file to another. It is what lets configuration 3 find L5.

Joern takes the idea all the way: it overlays the tree, the control flow and the data dependencies into a single graph, the code property graph (Yamaguchi and co-authors, 2014). Finding a taint flaw then means finding a path in that graph, from source to sink, that crosses no sanitizer. The richer the representation, the further the tool sees, and the more it costs in memory and time.

Taint, the long version

A taint engine needs four lists, and their quality matters as much as the engine itself:

Propagators are the forgotten ones: if the tool does not know that " ".join(pieces) carries the ink from the list to the text, the path stops dead, and the flaw vanishes from the report without a sound.

Then come the sensitivities: what the engine agrees to tell apart. Each one removes false alarms, and each one has a price.

x = request.args["x"]
x = "fixed"
db.execute(x)

u.name = request.args["n"]
u.role = "customer"
db.execute("… " + u.role)

a = clean(request.args["q"])
b = clean("SELECT 1")
db.execute(b)

if sort not in ("name", "price"):
    abort(400)
db.execute(f"… ORDER BY {sort}")

Key takeaway:“we do taint analysis” means almost nothing. Ask which of these sensitivities the tool handles, and which lists it relies on for your frameworks.

Why no SAST tool is perfect (and never will be)

This is not a lack of talent on the vendors' side, it is mathematics.
Rice's theorem (1953) says that no program can decide, for every program, a non-trivial property of its behaviour. “Can this program send an input into an SQL query?” is one.

So every tool has to choose which way to be wrong:

In practice, industrial tools mix both depending on the spot. Researchers even coined a word for it, soundiness (“In Defense of Soundiness”, Livshits and co-authors, 2015): rigorous everywhere, except where rigour would cost too much, for example around eval or reflection.

Key takeaway:a SAST tool guaranteed correct for all programs and their behaviours would solve the halting problem. If its vendor really had one, they would not be on a trade show stand, they would be collecting the Turing Award.

What breaks tools in real life

Our booby-trapped application fits in three tidy files. Real code is full of things that make the tool lose track of the ink:

• Frameworks.In Flask, inputs are called request.args; in Django, request.GET; in Spring, a parameter annotated @RequestParam. Without a model of the framework, the tool does not even know where the sources are: framework models are the real product.
• Dependency injection.In Spring, the object actually called is chosen at startup, from configuration. The call graph is then guessed, not read.
• Reflection and dynamic code.getattr(obj, name)(), eval, computed imports, decorators that wrap a function: all calls whose target can be hard, or impossible, to work out without running the code.
• ORMs and templates.Product.objects.filter(name=q) is parameterized; Product.objects.raw("…" + q) is not. Jinja2, in Flask, escapes HTML by default in .html templates, but a |safe or a Markup() switches that protection off. The tool has to know every variant.
• What it does not read.Generated or minified code, an unsupported language, a file that is too big, syntax too recent for the parser: often, the tool moves on silently.

Language models help read a path (in 2024, Google's Big Sleep found a memory flaw in SQLite before release), but they can invent a protection: a “false positive” verdict without proof is worth nothing. The right split: the engine provides the path, the AI helps read it, a human decides, and the decision is written down.

Wrapping up

Three tools, three lists: the gap does not come from the number of alerts, but from what each engine can model (the path from one file to another, the context of a sanitizer, the developer's checks) and from what it admits it did not read. None will ever be perfect, Rice sees to that. Prefer the one that shows the full path of every alert and says clearly where it stopped: how many files were analyzed, how many were skipped, and why.

To choose a tool (market engines, benchmarks, questions to ask): Choosing a SAST tool without being fooled. To revisit the basics: SAST explained simply. And for the blocks your code imports: SCA: from inventory to priority.

Read next: Choosing a SAST tool without being fooled