Architecture

Triage and Grading

Scanners report one finding per affected location, which is right for detection and unreadable as a report. Phase 9.5 shrinks the list by rule, asks an LLM what each survivor would cost and how easily it could be exploited, lifts the severity of anything both financial and live, and writes the summary a site owner will actually read.

Phase 9.5Stage 0 · no LLMStages 1–2 · LLMA–F grade

Stage 0: four passes, no LLM

Deduplication is a rule about where a fix goes, so it needs no model and costs nothing. It used to sit behind the LLM triage phase, which meant a scan run with --llm none shipped the raw list: 296 findings on Juice Shop, of which 272 were five issues restated once per endpoint. It now runs first, with or without an API key. The example below is thirteen findings; follow the right-hand counter.

raw findings · one per location Missing Content-Security-PolicyGET /kept · lists 4 endpoints Missing Content-Security-PolicyGET /api/products Missing Content-Security-PolicyGET /api/users Missing Content-Security-PolicyGET /rest/basket IDOR: object reachable by idGET /api/tasks/42INSTANCE · stays IDOR: object reachable by idGET /api/orders/7INSTANCE · stays Hardcoded Stripe secret keyconfig.ts:12kept · higher severity Stripe key in git historyconfig.ts:12 stripe_live key in plaintextlib/pay.ts:4kept · also affects: 2 more stripe_test key in plaintextlib/pay.ts:9 stripe_webhook secret in plaintextlib/pay.ts:15 Vulnerable jsonwebtoken 8.5.1 (CVE)package.json:31kept · 67% word overlap Vulnerable package jsonwebtoken 8.5.1package.json Measured on Juice Shop, --llm none, before dedup moved ahead of the LLM: 296 findings shipped · 272 of them were 5 issues restated per endpoint Stage 0 now runs first, so the no-key scan gets the readable list too. raw 13 pass 1 · exact title (INSTANCE: per location) 10 pass 2 · same file + line 9 pass 3 · scanner + category + file, 3+ 7 pass 4 · fuzzy title, word overlap ≥ 60% 6 no LLM call · no cost · runs with --llm none

1Thirteen findings arrive, one per location. Four scanners reported the same missing header on four URLs, two IDORs share a title, a Stripe key was found twice on one line and three times in one file, and two dependency scanners described the same package differently.

2Pass 1: exact title. Lower-cased, trimmed titles group together. The group keeps its best finding (highest severity, then an LLM-reviewed one, then the longest description) and appends "Affects 4 endpoints: …". Categories whose fix is per location are keyed by title plus location, which is why the two IDORs are not a group.

3Pass 2: same file and line. Two scanners, the secret scanner and the git-history scanner, flagged config.ts:12. One source line is one fix, so the best of the two is kept. For per-location categories the endpoint is part of the key, so a shared handler reached from two routes stays two findings.

4Pass 3: same SAST scanner, category and file, three or more. Three Stripe values in lib/pay.ts become one finding that lists the others. For per-location categories the title is part of the key, so SQL injection, command injection and XSS in one file stay three findings. This pass only applies to the deterministic SAST scanners; two LLM findings in one file are usually different issues and are left alone.

5Pass 4: fuzzy title. Titles are reduced to words of three or more letters minus stop words; both need at least three. Overlap is measured against the larger set, and 60% or more merges (90% for shared-root categories such as dependencies and secrets, whose titles differ in exactly the package or key type; titles in different categories never merge). "vulnerable, jsonwebtoken, cve" against "vulnerable, package, jsonwebtoken" is two of three. Two live findings on different endpoints never fuzzy-merge, and per-location categories never merge across locations.

6Six findings leave Stage 0, each still pointing at every place it was seen. The pass that produced the Juice Shop number was not a smarter rule; it was running the rules at all when no LLM key is present.

Remediation scope decides how a pass behaves

RemediationScope answers one question per category: how many places does the fix go? Every pass consults it, the fuzzy pass included, because that one is the loosest and a wrong merge there is both most likely and least visible.

SERVERone config change covers everything reported once · endpoints listed as evidence missing headers · CORS · mixed contentsource maps · SRI INSTANCEeach occurrence is its own fix every one reported · never collapsed IDOR · injection · auth · privilege escalationbusiness logic · open redirect · (default) SHARED_ROOTone cause, many call sites grouped under the cause · sites listed exposed secrets · dependenciesRLS · client exposure Treat everything as INSTANCE: one header, 76 times Missing Content-Security-Policy GET / Missing Content-Security-Policy GET /api/products Missing Content-Security-Policy GET /api/users Missing Content-Security-Policy GET /rest/basket … 72 more identical lines IDOR GET /api/tasks/42 the one finding that matters, on page four Treat everything as SERVER: four IDORs become one IDOR · GET /api/tasks/42 IDOR · GET /api/orders/7 IDOR · GET /api/users/3 IDOR · GET /api/files/9 IDOR · 1 findingthree ownership checks unwritten a report that looks handled and is not

1A category's scope is set once in code. Scanners keep reporting per location; the scope tells each dedup pass whether those locations are one problem or many.

2SERVER. Missing headers, CORS, mixed content, source maps and SRI are one setting on one server. Reported once, and the list of endpoints it was seen on is kept as evidence: 76 URLs says "the server has no CSP", which is what makes the single fix worth doing.

3INSTANCE. IDOR, injection, auth, privilege escalation, business logic and open redirect are each somebody writing a check in a specific place. Every occurrence is kept and never collapsed across locations. A category with no scope assigned defaults here, the safe direction.

4SHARED_ROOT. Exposed secrets, dependencies, RLS and client exposure have one cause reached from many places: rotate the key, bump the package, write the policy. Grouped under the cause, with the sites listed.

5Failure mode one. Without the scope, dedup on title alone was the only option, and turning it off meant "Missing Content-Security-Policy" restated on 76 endpoints. The real finding is on page four.

6Failure mode two. Dedup on title alone does the header case correctly and this one silently: four IDORs on four endpoints share a title, so three of them vanish behind something that looks handled. The two modes point opposite ways, which is why one rule cannot serve both.

Stage 1: impact × likelihood → priority, then calibration

Findings that still need judgement go to the LLM in batches of twenty, up to three batches at a time. For each one it returns an impact category and a likelihood level, and the priority is read from a fixed table. Findings from the deterministic SAST scanners never make the trip: their impact comes from a category table and their likelihood from their severity.

ACTIVELYEXPLOITABLE REQUIRESAUTH REQUIRESADMIN THEORETICAL FINANCIAL DATA_BREACH LEGAL OPERATIONAL REPUTATIONAL 1 2 3 3 1 2 3 3 1 2 3 4 2 3 3 4 2 3 4 4 priority 1 is highest, 4 lowest · a pair not in the table → 3 red cells: a HIGH finding landing here is escalated to CRITICAL, priority 1 Stage 1 · LLM enrichment findings needing judgement, batches of 20 up to 3 batches in flight the model returns, per finding: impact · likelihood · remediation priority = table[impact][likelihood] SQL injection POST /rest/products/search severity HIGH, from the scanner impact: data_breach likelihood: actively_exploitable → priority 1 Admin settings form lacks CSRF token POST /admin/settings severity MEDIUM, from the scanner impact: operational likelihood: requires_admin → priority 3 Server header reveals version GET / severity LOW, from the scanner impact: reputational likelihood: theoretical → priority 4 SAST-only scanners skip the LLM route_auth · secret · dependency · docker · iac … impact from a category table, e.g. exposed_secrets → data_breach business_logic → financial unencrypted_pii → legal likelihood from severity: CRITICAL → actively_exploitable HIGH, MEDIUM → requires_auth LOW → theoretical Stage 1.5 · calibration SQL injection · severity HIGH impact data_breach ✓ · actively_exploitable ✓ HIGH → CRITICAL · priority 1 only HIGH is lifted; MEDIUM never jumps, the model's own "not HIGH" is respected

1The table is fixed in code. Rows are the five impact categories, columns the four likelihood levels, cells the priority. The LLM's job is to place each finding; the priority is never asked of the model.

2A SQL injection on a public search endpoint: the model says data_breach and actively_exploitable. The cell reads 1, the highest priority.

3A missing CSRF token on an admin-only form: operational impact, exploitable only with an admin already logged in. Priority 3.

4A version string in the Server header: reputational, theoretical. Priority 4. Any pair the table does not list falls back to 3.

5Deterministic SAST scanners are placed by rule, with no LLM call: impact from the category (unknown categories count as operational), likelihood from severity. Anything the LLM batch skipped or timed out on gets the same rule-based placement, so nothing leaves triage without a priority.

6Stage 1.5 calibration. After enrichment, any finding still rated HIGH whose impact is financial or data_breach and whose likelihood is actively_exploitable is lifted to CRITICAL with priority 1. The two red cells are the only ones that trigger it. MEDIUM findings are never jumped.

Stage 2: the owner summary and the grade

One small LLM call receives the severity counts and a compact serialization of the findings and returns a summary for someone who does not read code: a plain-language risk statement, a letter grade, up to five key risks, and a remediation plan in phases. The headline grade on the report itself is computed by rule from the severity counts, on the ladder shown on the right.

Owner summary · Stage 2 one LLM call · input: scan mode · targetcritical / high / medium countscompact JSON of the triaged findings asked back, as JSON: risk_summarygrade A | B | C | D | Fkey_risks[] → cut to 5remediation_phases[] if the call fails: count-based fallback grade, no risks, no phases FC+ Anyone with a browser can read other customers' orders. Two issues need fixing before launch. illustrative wording · the grade letter is the model's, A to F Key risks · capped at 5 1 Anyone can read other users' orders 2 Product search accepts SQL 3 A Stripe key is committed to git 4 The admin form has no CSRF token 5 jsonwebtoken 8.5.1 is outdated Remediation plan · each finding in exactly one phase Phase 1 · 3immediateCRITICAL + HIGH Phase 2 · 2short-termMEDIUM Phase 3 · 4ongoingLOW Report grade · calculate_grade, first match from the top wins Fany criticalfix before launch1 critical ✓0 ✗ Dany highfix before launch0 ✗ C3 or more mediumfix soon2 < 3 ✗ C+1 or 2 mediumfix soon2 medium ✓ B6 or more lowgood, work through them B+3 to 5 lowgood A-1 or 2 lowvery good Ano findingssafe to ship A+no findings and hardenedsafe to ship A = safe to ship · C = fix soon · F = fix before launch example: 1 critical · 2 high · 2 medium · 4 low → F fix the critical and both highs: 0 · 0 · 2 · 4 → C+

1The prompt carries the scan mode, the target, the totals of critical, high and medium findings, and a compact JSON of the triaged list. The model is asked for a fixed JSON shape in return.

2Risk summary and grade. A sentence or two a site owner can act on, and a letter from A to F. The letter here is the model's; if the call fails, a count-based fallback grades A with no critical or high, B with at most two high, C with at most five, D with at most one critical and ten high, F beyond that.

3Key risks. Whatever the model lists is cut to five. Each is meant to be readable without the finding it came from.

4Phased plan. The model names the phases; the counts are filled by rule so they always add up: CRITICAL and HIGH findings in phase 1, MEDIUM in phase 2, LOW in phase 3. Each finding belongs to exactly one phase.

5The report grade does not come from the model. calculate_grade walks the ladder from the top and stops at the first rung that matches: any critical is F regardless of everything else. The example counts land there. If a scanner timed out or crashed (recorded in scanner_errors), the verdict reads "scan incomplete" and the grade is labelled provisional, unless critical or high findings already decide it.

6Fix the critical and both highs and the same ladder gives C+: no critical, no high, two medium is fewer than three. The report also lists these steps as "what it takes to reach the next grade", computed from the same thresholds.

Why it matters

The raw list is the wrong unit

A scanner is right to emit one finding per URL: that is where it saw the problem. A report built from that list is wrong twice over. Collapse by wording and three of four IDORs disappear; refuse to collapse and one missing header fills four pages. The scope rule exists because those two mistakes point in opposite directions, and the measured Juice Shop list, 296 findings of which 272 were five issues restated per endpoint, is what the first mistake looks like at scale.

Severity is not priority

Severity says how bad the bug class is. Priority says what to fix this week, which depends on who can reach it and what it costs the business. The matrix makes that a table lookup, and calibration catches the one combination, live and financial or data-exposing, where a HIGH label would understate a breach already in reach.

Limits and exact rules

WhereRuleEffect
Pass 1 keep ruleHighest severity, then a finding from llm_code_reviewer, then the longest description.The surviving wording may be the LLM's even when a rule scanner found it first.
Pass 3Only for the deterministic SAST scanners (route_auth, express_middleware, drizzle_schema, iac, docker, shell_script, dependency, firebase_rules, secret, static_injection). Groups of one or two are left alone.LLM findings in the same file are never grouped by this pass.
Pass 4Words of three or more letters, stop words removed, at least three words on both sides; overlap against the larger set must be 60% or more. Never across two different live endpoints; never across locations for INSTANCE categories.Short titles and single-word differences do not fuzzy-merge.
Scope defaultA category not in the scope table counts as INSTANCE.Unmapped categories are never collapsed; they may repeat.
Stage 1Batches of 20, up to 3 in flight. SAST-only scanners are placed by rule. Unknown category → operational impact; unknown pair → priority 3.Rule-placed findings never get the model's narrative, only the category remediation text.
CalibrationOnly HIGH is lifted, only for financial or data_breach impact with actively_exploitable likelihood.A MEDIUM the model under-rated stays MEDIUM.
Owner summaryKey risks cut to 5. On LLM failure the fallback summary has a count-based A–F grade and empty risks and phases.A report without an API key still has a report grade from the ladder, with no narrative.
Without an LLMOnly Stage 0 runs. Findings keep scanner severity; the ladder grade is still computed.No impact, likelihood, priority or themes in the output.