Endpoint Discovery
A scanner can only test an endpoint it knows about. This page follows one target from its first HTML response to a ranked inventory: which files get read, which paths count as attack surface, where methods and parameters come from, and what an authenticated crawl adds.
Reading a code-split single-page app
The served HTML of a single-page app is a shell. The routes and the API calls live in JavaScript, and a bundler splits that JavaScript into an entry bundle plus lazy chunks. Which of those files the scanner fetches decides what the rest of discovery can see.
1Playwright renders the target and keeps the HTML. For a single-page app that HTML is a shell: an empty <div id=app>, a few <script src> tags, and a list of <link> tags that preload the lazy route chunks.
2A script-only sweep fetches what <script src> names: on Juice Shop, three files. The route chunks are referenced only by <link rel="modulepreload|preload|prefetch">, so nothing fetches them.
3The gap: 10 of Juice Shop's 13 JavaScript files never reached the regexes, and with them /api/Users and /api/Feedbacks. A scanner cannot test an endpoint nothing told it about.
4The fix matches on the href rather than on a list of rel values: a <link> whose href ends in .js or .mjs exists to load that JavaScript. Scripts and chunks are collected in document order, de-duplicated, so the asset cap truncates the tail of the page rather than all of the chunks.
5Url-only discovery on Juice Shop went from 61 endpoints to 123, with no change to the regexes themselves.
Routes vs endpoints inside a bundle
A bundle names two kinds of path. The ones the app's own router renders are pages; the ones the app sends requests to are endpoints. Only the second is attack surface, and the difference is visible in how each is written.
1Two lines from a minified bundle. {path: "/about", component: About} is a route the app's own router renders. new Es({url: Z.hostServer + "/file-upload"}) is an endpoint the app sends a request to.
2The route regex matches any quoted path of three or more characters. For each match the scanner looks back 60 characters for a request-making token: fetch, axios, XMLHttpRequest, .open, .get/.post/.put/.patch/.delete, url:, baseURL, hostServer, apiUrl, endpoint. Before "/about" there is only {path:, so it is dropped.
3Before "/file-upload" sit url: and hostServer. Kept, as GET /file-upload with source requested_path. The window is deliberately short: widening it starts catching the previous statement's call.
4This replaced a list of interesting-looking directory names. On Juice Shop that list kept 11 of 55 paths and discarded /file-upload, /dataerasure, /profile and /data-export. The filter now reads how a path is used, which needs no vocabulary.
5Four file-upload vulnerabilities sit behind /file-upload. With the path dropped they were unreachable no matter how good the file-upload scanner was; with it kept, it enters the inventory and the ranking in section 06.
# a path was kept if it started with a word someone thought of
INTERESTING = ("/dashboard", "/api", "/apps", "/admin", "/account")
for route in re.findall(ROUTE_PATH_PATTERN, content):
if not route.startswith(INTERESTING): # /file-upload: dropped
continue
add(route)# constants.py: a path is worth probing when the bundle makes a request to it
REQUEST_CONTEXT_PATTERN = (
r"(?:fetch|axios|XMLHttpRequest|\.open|\.get|\.post|\.put|\.patch"
r"|\.delete|url\s*:|baseURL|hostServer|apiUrl|endpoint)"
)
REQUEST_CONTEXT_WINDOW = 60 # wider starts catching the previous statement's call
# endpoint_discovery.py
for match in re.finditer(ROUTE_PATH_PATTERN, content):
preceding = content[max(0, match.start() - window):match.start()]
if context.search(preceding): # url: Z.hostServer + "/file-upload"
add(route, source="requested_path")Seven regex families over the bundle
The HTML and every fetched JavaScript file are concatenated and run through seven families of patterns. Each match is resolved against the target, filtered, and stored under a METHOD:url key so the same endpoint found twice is kept once.
1fetch. fetch("url") is a GET. fetch("url", {method: "POST"}) carries its method, which is the only place a bundle states one explicitly. A method the map does not know falls back to GET.
2axios. axios.get/post/put/patch/delete("url"): the method is the function name.
3XMLHttpRequest. .open("METHOD", "url") gives both the method and the path.
4API path literals. Any quoted path under /api, /rest, /graphql or /v1, plus the same shapes inside a template literal after a ${…} interpolation. Both become GET for now.
5Supabase. .from("table") becomes GET /rest/v1/table, .rpc("fn") becomes POST /rest/v1/rpc/fn. The project URL and anon key found in the same bundle are used later to read the project's own OpenAPI listing of tables.
6Parameterised paths. /users/:id and /items/{id} declare a path parameter, so the endpoint is created with has_path_params set and the parameter named. IDOR and per-parameter injection need exactly that.
7Route definitions in request context, the pass from section 02. Every match goes through the same gate: _resolve_url drops unresolved template literals and resolves relative api/, rest/, graphql/, v1/ paths against the target; _should_include drops static assets, /_next/-style prefixes and known third-party hosts.
Spec, forms and probing: the three other sources
Regexes over a bundle find what a JavaScript frontend calls. A REST API with no frontend, a server-rendered app with no bundle, and an API base no bundle mentions each need their own source. All four feed the same inventory.
1OpenAPI. For each API base the scanner tries the well-known spec locations (13 per the architecture doc). A 200 whose body parses as JSON with a paths object yields every declared operation with its real method, path parameters including {templated} segments, and query parameters. This is how a REST API with no browsable frontend becomes testable.
2HTML forms and links. A standard-library HTMLParser turns each <form action> into an endpoint at its resolved action, GET or POST by the form's method, with its named input/select/textarea fields as parameters; submit, button and hidden inputs are skipped. An <a href> counts only when it carries query parameters. If JS and spec discovery found fewer than 10 endpoints, a bounded same-origin crawl of up to 15 pages follows links to more forms.
3Active probing. Common API paths such as /api and /graphql are requested on the target origin and on any api./backend. host found in the bundle. A probe is kept when the status is below 400 and the body is JSON by content type or by its first character. A Supabase project URL is probed at /rest/v1/, whose OpenAPI listing names every exposed table.
4Everything merges under a METHOD:url key. Then each GET collection such as /api/Products gets an object-level twin /api/Products/1 with a path parameter named id. Only after that are parameters detected and categories assigned.
The authenticated crawl
With credentials, a headless browser logs in through the real login form and walks the app behind it. This adds the endpoints that only appear after login, the session every later scanner reuses, and the ids the authenticated user owns.
1With credentials, Playwright opens the login page. It has more than one text input: a search box in the header and the login form itself. A hardcoded selector such as input[name=email] finds nothing here, because the identity field is called userName.
2Form-scoped detection: find the visible password field, scope to its enclosing <form>, and take the first visible text/email/tel input in that same form as the identity field, whatever it is named: email, userName, login, handle. The search box sits outside the form, so it is never grabbed.
3Both fields are filled through real input events so framework-controlled inputs register, the submit button is clicked, and the session is read back: a token from browser storage, or the auth headers seen on intercepted calls. That session is handed to every auth-aware DAST scanner afterwards.
4Every response is intercepted. URLs that look like API calls are kept with their request headers and body, status, a 2000-character response preview, and the ids found in the body: UUIDs anywhere, plus JSON keys named id or ending in _id.
5The crawler walks pages breadth-first from a queue seeded with the app's own routes and common authenticated paths, up to 50 pages and 30 links per page. The HTML form extractor from section 04 runs after each page load, because a server-rendered form makes no network call to intercept.
6Ids are aggregated per path, or per Supabase table, as the authenticated user's owned resource ids. The IDOR scanner later asks whether User B can fetch User A's.
Categorise, then rank per attack dimension
Every endpoint gets a semantic category from its path. Before the scanners run, a shared prioritizer scores the inventory for the vulnerability class each scanner cares about, so what a time budget reaches first is the endpoint most likely to carry that bug.
1One category per endpoint, first matching rule wins: /auth /login /signup /oauth → AUTH, /admin → ADMIN, /user /profile /account /me → USER_DATA, /file /upload /download /media → FILE_ACCESS, /payment /invoice /billing /charge → PAYMENT, an id parameter → RESOURCE_CRUD, /health /status /version /ping → PUBLIC, otherwise UNKNOWN.
2Before the scanners run, endpoint_prioritizer.rank() scores each endpoint for one dimension. For INJECTION: a query string +3, a path parameter +2, a search/query/find/filter/lookup/list word in the path +3, a body-carrying method +1. The sort is stable, so ties keep discovery order.
3A file upload takes no query string, no path parameter and no body field; its payload is the file. Every parameter-shaped signal scored it zero. On Juice Shop /file-upload ranked 73rd of 77, below the old cap of 30, so the two XXE bugs behind it were unreachable however good the probe was.
4The rule: FILE_ACCESS counts +3 toward INJECTION. /file-upload moved to 6th of 77. No app-specific string is involved; the category came from the path word /upload.
5The other dimensions use the same kind of generic signal. The injection, XSS, IDOR, CSRF, auth-bypass and HTTP-probe scanners all take their order from this one function, and that order decides what a time budget reaches first.
--probe-writes: reaching the mutation surface
A path in a bundle carries no HTTP method, so everything the regexes find is a GET. Any scanner that tests a POST or PUT then has nothing to work on. --probe-writes derives the state-changing methods from REST shape, at the cost of writing to the target.
1Url-only discovery produces GET endpoints only, because a path in a bundle carries no method. The scanners for stored XSS, mass assignment, CSRF and SSTI filter on POST/PUT and find nothing to test.
2--probe-writes derives state-changing twins from REST shape: POST to a collection, PUT to an item, the same convention that already produced /api/Products/1. DELETE is derived too and deliberately never emitted. Each twin carries source rest_method; the inventory roughly doubles.
3It is off by default. The scan now writes to whatever it is pointed at: it creates records, and an endpoint that sends email or charges a card will do so. A scanner that deletes a customer's record to prove it could is not worth the finding.
4Measured on Juice Shop v20.1.1, url-only. The benchmark (RESULTS.md, README, architecture doc) reports 22/45 → 28/45 and ~72 minutes against 27. What was gained: CSRF, three XSS challenges, one IDOR and mass assignment; nothing was lost.
5The first measurement did not look like that. It read six gained, six lost, and the six lost were exposed_data and info_disclosure, both scored off http_probe. That scanner was running 901 s against a 900 s timeout, and the hard cancel discarded every finding it held. The doubled inventory had pushed it over.
6One bug wearing the costume of a trade-off. With scanners checking budget.expired() in their per-endpoint loop and returning early, the measurement reads six gained, nothing lost. The mechanism is animated on the scan pipeline page.
Coverage limits: time is the bound, not a count
Scanners used to stop after a fixed number of endpoints. Against an app with 77 endpoints those caps decided which quarter got examined, in ranked order, and nothing said so.
Probing costs HTTP requests rather than tokens. Measured over nine scans, a DAST run cost $0.12 to $0.18 whether it covered 59 endpoints or 154, while a SAST run is a flat ~$1.28 because its cost is the LLM reading files. The endpoint was never the scarce thing, so the caps of 20 for XSS, 30 for injection and 5 for CORS were removed. MAX_ENDPOINTS_TO_DISCOVER is now unbounded.
Time, per scanner, through TimeBudget, with the per-scanner timeouts raised so that timing out means something is wrong rather than that the app was large. Every scanner that walks endpoints emits per-endpoint progress, so a long run is legible rather than silent. A scan that takes half an hour and finishes beats one that takes ten minutes having skipped two thirds of the surface.
| cap | bounded | status | why |
|---|---|---|---|
XSS: 20 endpoints | which endpoints got tested | removed | HTTP requests are not scarce; ranking plus time decides coverage |
injection: 30 endpoints | which endpoints got tested | removed | this cap is what hid /file-upload at 73rd of 77 |
CORS: 5 endpoints | which endpoints got tested | removed | same |
MAX_ENDPOINTS_IN_PROMPT = 50 | endpoints listed in the business-logic LLM prompt | kept | LLM tokens are genuinely scarce |
MAX_OOB_POST_ENDPOINTS = 10 | POST endpoints registered for out-of-band callbacks | kept | registrations against an external callback service are genuinely scarce |
run_scanner_safe cancels a scanner at its hard timeout, and a cancelled coroutine cannot hand anything back. Scanners therefore stop themselves: the runner publishes its deadline, a TimeBudget() with no argument inherits it, and the check sits in the per-endpoint loop rather than between phases, because one phase can walk the whole inventory. http_probe runs its fixed-cost checks before the ones that scale with the inventory, so a large app cannot spend its whole budget on method tampering and never reach the check that finds an exposed .env. The lane-by-lane animation is on the scan pipeline page.