# Trust risk signals: method and measured error rates

> Signals only; a site with low observed risk can still be malicious. Not a safety certification.

Applies to the quick checks url-trust, site-check (its trust part), site-verdict, verdict-card and domain-lookalike, output format `trust/2`. None of them ever answers "safe" or "legit". They report:

- `signals[]`: each signal with its id, the observed value, evidence (URL, header, TLS certificate, DNS or RDAP record, HTML snippet, timestamp), direction (raises / lowers risk), weight and basis (observed / inferred).
- `risk`: `level` in `low_observed_risk`, `elevated`, `high`, `unknown`; `score` 0-100 where higher means MORE observed risk (not a probability); `confidence` low / medium / high; `observed_bad_rate`: the share of known-bad sites among evaluation targets with a similar score, with a 95% interval.
- `coverage`: what was checked, what failed and what was not checked; `not_a_guarantee: true`.
- Deprecated for one version: the old `verdict` field (and site-check `overall.trust_verdict`) now carries the same value as `risk.level`; removed after 2027-01-31. The old values safe / likely_legit / looks_normal are never returned.

## Signals used

- **url-trust**: One page fetch with redirects, TLS certificate, RDAP registration date, hostname analysis (brand lookalike, punycode / mixed scripts, brand outside its domain, phishing words, risky TLD, shape), page content (parked / for-sale, scam phrases, password form on a lookalike, seed-phrase form), security-header grade and, for HTTP 402, the x402 challenge. Each finding adds risk by severity (critical 45, high 20, medium 8, low 3); a critical finding forces level high.
- **site-check**: The trust part of site-check IS url-trust (same function, same output under "trust"); its error rates are the url-trust rows.
- **site-verdict**: One page (contact details, address, policies, company-registration text, structured data, copyright year, parked / placeholder / scam text), TLS, RDAP age, MX / SPF / DMARC, then the local model (qwen3, on our own machine, no cloud AI) reads the page text and the signals. Risk = 100 - (0.6 x rule score + 0.4 x model score), with caps for parked pages, scam phrases, no valid HTTPS and very thin pages.
- **verdict-card**: The site-verdict result drawn as an SVG card; same risk and error rates as site-verdict.
- **domain-lookalike**: Offline string analysis of the host name only (no DNS, no web request): brand lookalike, brand outside its official domain, mixed alphabets, punycode, risky TLD, phishing words, hyphens, subdomain depth.

## What we do NOT check

- No third-party blocklist, reputation feed or threat-intelligence service is consulted.
- Only the page that was fetched is read; no other pages, no JavaScript is executed, no form is submitted, nothing is logged into.
- Point in time: a site can change minutes after the check (phishing kits are often switched on later, or only for some visitors, countries or user agents).
- Company registration numbers, addresses and reviews are detected as text, not verified against registries.
- Header and TLS observations come from one network vantage point.

## Evaluation data

Results file: `results-2026-10-30.json` (harness `eval/trust/run.mjs`, created 2026-10-30). Snapshot recorded 2026-10-30T07:01 to 2026-10-30T14:41 UTC; every target fetched once through the same guarded fetchers the paid tools use, at most one HTTP request per second, then replayed offline.

- **CERT Polska Warning List** (known-bad): https://hole.cert.pl/domains/v2/domains.json. Terms: https://cert.pl/en/warning-list/ (checked 2026-10-30): Any purpose including commercial; no warranty; do not imply endorsement by NASK-PIB / CERT Polska. File sha256 d7b59c9db6d3806e...
- **Majestic Million (2026-10-30 download)** (known-good): https://majestic.com/reports/majestic-million. Terms: https://majestic.com/reports/majestic-million (checked 2026-10-30): Creative Commons Attribution 3.0 (CC BY 3.0): commercial use allowed with attribution. Attribution: Majestic Million, Majestic-12 Ltd (majestic.com), CC BY 3.0. File sha256 a5bbc145c8e4d780...
- Not used: OpenPhish Community feed: Terms forbid commercial purposes and research that supports an organisation's business operations.
- Not used: URLhaus (abuse.ch): Bulk lists need an account Auth-Key; commercial use may need a paid subscription. Also malware-distribution URLs, not phishing.
- Not used: PhishTank: New registrations closed; terms document (Cisco) not confirmable without an account.

Targets by bucket: phish_certpl 300 (reachable 39); phish_certpl_fresh3d 237 (reachable 62); top_1k 150 (reachable 107); long_tail_100k_1m 100 (reachable 71); mid_1k_100k 150 (reachable 100).
Known-bad sites that were unreachable or answered an HTTP error at recording time (taken down) are reported separately and never counted as correct detections. Known-good sites that could not be fetched (API hosts, CDNs, bot walls) are also excluded from the rates.

Thresholds and confidence bands were fitted on one half of the reachable targets (seeded split 130, fit n=180) and every rate below is measured on the other, held-out half (n=199).

## Measured error rates (held-out half)

| Tool | Thresholds (elevated / high) | False positives: good flagged elevated or high | False negatives: live bad rated low_observed_risk | Precision of "elevated or high" | Recall | Rated unknown (of reachable targets; unreachable ones are excluded above) |
|---|---|---|---|---|---|---|
| url-trust | 9 / 44 | 6.6% (95% CI 3.5%-12.0%; 9/137) | 38.7% (95% CI 27.6%-51.1%; 24/62) | 80.8% (95% CI 67.5%-89.6%; 38/47) | 61.3% (95% CI 48.9%-72.4%; 38/62) | 0.0% (95% CI 0.0%-1.9%; 0/199) |
| site-verdict | 36 / 63 | 5.5% (95% CI 2.7%-10.9%; 7/127) | 37.0% (95% CI 24.5%-51.4%; 17/46) | 80.6% (95% CI 65.0%-90.3%; 29/36) | 63.0% (95% CI 48.6%-75.5%; 29/46) | 0.0% (95% CI 0.0%-2.2%; 0/173) |
| domain-lookalike | 15 / 40 (defaults kept, see notes) | 0.7% (95% CI 0.1%-4.0%; 1/137) | 100.0% (95% CI 94.2%-100.0%; 62/62) | 0.0% (95% CI 0.0%-79.3%; 0/1) | 0.0% (95% CI 0.0%-5.8%; 0/62) | 0.0% (95% CI 0.0%-1.9%; 0/199) |

Rates are k/n with Wilson 95% intervals. Precision depends on how many bad sites are in the mix, so it is not your precision.

- domain-lookalike: kept defaults: fit half has 39 live known-bad, 141 known-good, AUC 0.518 (need >= 10, >= 30, >= 0.6).
- domain-lookalike, separate measurement on all known-bad NAMES including domains already taken down (it reads the name only): false negatives 100.0% (95% CI 98.7%-100.0%; 289/289), recall 0.0% (95% CI 0.0%-1.3%; 0/289), false positives 0.7% (95% CI 0.1%-4.0%; 1/137).

## By bucket (all reachable targets, fitted thresholds)

| Tool | Bucket | n | low_observed_risk | elevated | high | unknown |
|---|---|---|---|---|---|---|
| url-trust | phish_live | 101 | 38 | 61 | 2 | 0 |
| url-trust | top_1k | 107 | 105 | 2 | 0 | 0 |
| url-trust | long_tail_100k_1m | 71 | 62 | 6 | 3 | 0 |
| url-trust | mid_1k_100k | 100 | 91 | 8 | 1 | 0 |
| url-trust | phish_dead_answered | 111 | 0 | 0 | 9 | 102 |
| site-verdict | phish_live | 81 | 30 | 45 | 6 | 0 |
| site-verdict | top_1k | 99 | 97 | 1 | 1 | 0 |
| site-verdict | long_tail_100k_1m | 64 | 54 | 8 | 2 | 0 |
| site-verdict | mid_1k_100k | 96 | 89 | 6 | 1 | 0 |
| site-verdict | phish_dead_answered | 5 | 1 | 4 | 0 | 0 |
| domain-lookalike | phish_live | 101 | 101 | 0 | 0 | 0 |
| domain-lookalike | top_1k | 107 | 105 | 2 | 0 | 0 |
| domain-lookalike | long_tail_100k_1m | 71 | 71 | 0 | 0 | 0 |
| domain-lookalike | mid_1k_100k | 100 | 100 | 0 | 0 | 0 |
| domain-lookalike | phish_dead_answered | 436 | 436 | 0 | 0 | 0 |

## Score threshold curve (all reachable targets)

If "flagged" meant risk score >= t:

| Tool | t | good flagged (FPR) | live bad flagged (TPR) |
|---|---|---|---|
| url-trust | 0 | 100.0% | 100.0% |
| url-trust | 10 | 7.2% | 62.4% |
| url-trust | 20 | 5.8% | 52.5% |
| url-trust | 30 | 2.9% | 4.0% |
| url-trust | 40 | 2.2% | 2.0% |
| url-trust | 50 | 0.0% | 2.0% |
| url-trust | 60 | 0.0% | 0.0% |
| url-trust | 70 | 0.0% | 0.0% |
| url-trust | 80 | 0.0% | 0.0% |
| url-trust | 90 | 0.0% | 0.0% |
| url-trust | 100 | 0.0% | 0.0% |
| site-verdict | 0 | 100.0% | 100.0% |
| site-verdict | 10 | 66.8% | 98.8% |
| site-verdict | 20 | 33.6% | 79.0% |
| site-verdict | 30 | 25.1% | 71.6% |
| site-verdict | 40 | 3.5% | 56.8% |
| site-verdict | 50 | 1.9% | 49.4% |
| site-verdict | 60 | 1.9% | 14.8% |
| site-verdict | 70 | 0.8% | 3.7% |
| site-verdict | 80 | 0.0% | 0.0% |
| site-verdict | 90 | 0.0% | 0.0% |
| site-verdict | 100 | 0.0% | 0.0% |
| domain-lookalike | 0 | 100.0% | 100.0% |
| domain-lookalike | 10 | 1.1% | 7.9% |
| domain-lookalike | 20 | 0.7% | 0.0% |
| domain-lookalike | 30 | 0.7% | 0.0% |
| domain-lookalike | 40 | 0.0% | 0.0% |
| domain-lookalike | 50 | 0.0% | 0.0% |
| domain-lookalike | 60 | 0.0% | 0.0% |
| domain-lookalike | 70 | 0.0% | 0.0% |
| domain-lookalike | 80 | 0.0% | 0.0% |
| domain-lookalike | 90 | 0.0% | 0.0% |
| domain-lookalike | 100 | 0.0% | 0.0% |

ROC AUC (score only): url-trust 0.812, site-verdict 0.867, domain-lookalike 0.534.

## Notes from this run

- phish_certpl: 261 of 300 known-bad domains were unreachable or answered an HTTP error when recorded (taken down, or nothing at the root URL); they are excluded from the rates.
- phish_certpl_fresh3d: 175 of 237 known-bad domains were unreachable or answered an HTTP error when recorded (taken down, or nothing at the root URL); they are excluded from the rates.
- Owner decision 2026-10-30: the earlier known-good list (Tranco) includes a CC BY-NC input and Lordevo is a commercial company, so it is not used at all. Known-good targets were re-sampled from the Majestic Million (CC BY 3.0) and recorded again; the earlier known-good recordings were removed from the snapshot. The known-bad CERT Polska sample was drawn on 2026-10-30 from the 30-day pool after removing domains on the earlier top-sites list; it is reused as recorded. Known-bad targets on the Majestic Million are excluded from scoring. Excluded known-bad targets: 13.
- Bucket phish_certpl_fresh3d was added after the first recording showed most older entries already dead: Older Warning List entries were mostly unreachable at recording time; this batch samples entries inserted in the last 3 days of the same download.
- Known-bad = domains on the CERT Polska Warning List (mostly phishing aimed at Polish users); the root URL https://<domain>/ was checked, while the phishing page may sit on a deeper path.
- site-verdict refuses pages whose robots.txt disallows our agent or that answer HTTP errors; those are listed as not_scored and excluded from its rates.
- site-check is not run separately: its trust part is url-trust.

## Known failure modes

- Phishing on a compromised, long-established domain looks old, has valid TLS and good mail records: the domain-level signals lower its risk.
- Phishing pages that only show their content to some visitors (cloaking) look like ordinary or empty pages to us.
- Well-known sites with unusual setups (no HTTPS redirect, CDN error pages, bot walls, thin landing pages) can get elevated risk.
- The brand list for lookalike detection covers commonly phished brands only.
- The evaluation set mixes known-bad and known-good sites in proportions that are not those of real traffic, so precision in your traffic will differ.

## Reproduce

The harness is part of Lordevo's shop source tree (`eval/trust/`, not public): `node eval/trust/run.mjs` replays the frozen snapshot offline and rewrites the results file this page is built from; `--live-refresh` records a new snapshot (opt-in).

> Signals only; a site with low observed risk can still be malicious. Not a safety certification.