Recon output is not knowledge. It is a pile of names, URLs, scripts, status codes, and screenshots. I used to mistake the size of that pile for progress. A run that produced thousands of lines felt productive even when I could not say what had changed or what I should test next.

My current workflow has a stricter goal: reduce the surface to a short list of questions that can be checked by hand.

I do this only on assets I am authorized to test. Recon is not permission, and a hostname appearing in a certificate or archive does not automatically put it in scope.

Start with an inventory I can explain

The first pass is mechanical. I collect candidate hosts, check which ones answer, crawl in-scope pages, and preserve the raw output. BBHUNTER wraps this work in a 13-phase pipeline, but the individual tools are replaceable. The current state.json records aggregate completion and counts; it is not a full per-phase provenance log. I keep the raw phase directories because they are the only way to trace a lead back to its source.

bbhunter recon example.com

I keep the original observations before normalizing them. The current pipeline deduplicates hosts and URLs, but it does not attach a source and first-seen time to every row. That is a limitation I work around in my notes and want to fix in the tool.

The OWASP Web Security Testing Guide recommends identifying requests and parameters before testing. That sounds obvious, but it is the line between mapping an application and spraying guesses at it.

Crawl with boundaries

I use a crawler to learn how the application links its own surface together. Katana is useful here because it supports JavaScript parsing, headless crawling, and scope controls. Its official documentation also makes the two execution modes explicit: a standard crawler for ordinary HTML and a headless browser for applications that need a real browser to reveal routes.

I begin with the cheaper mode and move to headless only when the application needs it. I do not assume archived or third-party URLs are testable. They are references to review against the program scope.

For a host that the program explicitly allows, a small bounded run looks like this:

katana -u https://app.example.com -fs fqdn -jc -rl 5 -o recon/katana.txt

-fs fqdn keeps the crawl on that exact host, -jc parses JavaScript for endpoints, and -rl 5 holds the request rate below the tool default. I still follow the program’s own rate limits when they are lower.

The output I want is not “every URL on the internet related to this company.” It is:

host -> page -> script -> route -> parameter -> observed response

The current BBHUNTER pipeline does not preserve this full chain: it merges URLs from several tools, extracts JavaScript URLs, and downloads a capped set. The chain above is the provenance model I am working toward, not a feature I can claim today.

JavaScript is documentation written for the browser

Modern frontends often carry route templates, API clients, feature flags, error messages, and source-map references. The OWASP guide specifically calls out JavaScript, comments, metadata, and source maps as places where internal information can leak. It also warns that a source map may reconstruct human-readable source. See OWASP’s information-leakage review.

During manual review, I start with scripts observed on in-scope pages. The automated pipeline currently loses some page-to-script context when it merges crawler output. I search its downloaded files for concrete strings before asking a model to explain anything:

rg -n --no-heading \
  '/api/|/graphql|Authorization|Bearer |sourceMappingURL|WebSocket' \
  targets/<program>/recon/<timestamp>/js/files/

This produces noisy matches, which is fine. A route in a bundle is not an exposed endpoint. A string named admin is not an authorization bypass. The match is an address for reading.

For each promising client call, I record the same fields. This is a synthetic example, not a disclosed finding:

{
  "script": "<bundle>",
  "route": "<route>",
  "method": "POST",
  "parameters": ["format"],
  "auth_evidence": "Bearer token added by request wrapper",
  "observed_in_browser": false,
  "open_questions": [
    "Which role can call this route?",
    "Is the account inferred from the token or accepted as input?"
  ]
}

The false matters. It prevents extracted code from being confused with observed behavior.

Where I let a model help

I use a model after collection, not in place of it. I give it small, cited slices of evidence and let it sort and explain:

• group route variants without discarding the originals;
• explain an unfamiliar request wrapper;
• identify which parameters appear to select users, tenants, files, or redirects;
• list benign and risky interpretations of a response difference;
• turn observations into questions, each linked to its source file.

The model does not get to invent. I use this output shape:

For each item, return:
- observation: quote or exact value from the supplied artifact
- source: file and line, or request identifier
- hypothesis: one sentence
- benign_explanation: one sentence
- manual_check: a bounded test
- unknowns: facts not present in the evidence

Never invent a route, parameter, response, technology, or vulnerability.
Do not label an item a finding.

If the model cannot cite the supplied artifact, the item goes away.

Diffing beats repeated full runs

After the first inventory, I care more about change than volume. New hosts, new scripts, changed route sets, and different response shapes are where I look first. Many turn out to be build noise, cache churn, or renamed chunks; the diff gives me a place to start, not a reason to claim exposure.

I sort and deduplicate normalized snapshots under the same locale before comparing them:

LC_ALL=C sort -u raw/hosts.txt > current/hosts.txt
LC_ALL=C sort -u raw/routes.txt > current/routes.txt
LC_ALL=C comm -13 previous/hosts.txt current/hosts.txt
LC_ALL=C comm -13 previous/routes.txt current/routes.txt
git diff --no-index previous/scripts.sha256 current/scripts.sha256

The commands are simple on purpose. I want an auditable answer to “why am I looking at this?” A new subdomain is worth inspecting because it is new, not because an LLM described it as “high value.”

A hypothesis is still not a finding

A useful recon note connects an observation to a bounded test. This is the template, not a disclosed finding:

Observation: the client sends a tenant identifier in the export request.
Hypothesis: the server may trust that identifier instead of the authenticated tenant.
Benign explanation: the server may validate both values and reject a mismatch.
Manual check: use two owned test tenants and change only the identifier.
Stop condition: no access to data outside those controlled accounts.

This format forces me to state what would disprove the idea. It also makes dead ends useful: if the server validates the tenant correctly, I record that result instead of letting the same lead return during the next run.

What I keep out of the model

Scope decisions stay deterministic. Raw request and response capture stays in the proxy. Host and route diffs come from files. Secrets are redacted before any artifact is shared with an external service. A model never receives session cookies, private program material, or data that the program rules prohibit me from sending elsewhere.

I started with a pile of output. I stop when I have fewer tabs open and a written reason for opening each one.