Methodology
Scores (sar-score-2.0)
Seven components are normalized to 0-100 (adoption /30, trust /25, health
/20, ecosystem /15, domain fit /100, governance /100, docs /10) and weighted
by the domain’s profile in config/scoring.json. Two of them are never taken
from the curated workbook: trust is derived from the publisher identity
tier and the protocol probes, and governance from credited compliance
evidence (next section), for workbook and discovered rows alike. The
workbook’s five other values (adoption, health, ecosystem, fit, docs) are
kept because nothing better is known for them, and its original values stay
on every record as seed_components. Each domain’s Top 100 keeps the
workbook’s order (seed_rank): the curator’s ranking is the domain-fit
judgment that the components do not carry (the workbook’s fit value is 100
for nearly every row), and an experiment that sorted by the recomputed score
put general platforms first in every regulated domain on the strength of
their vendors’ certifications. The evidence-weighted score is shown beside
the rank, never used to sort it. Vendor-level evidence (a certification the
organisation holds, scope unknown for this agent) earns half credit in the
governance component.
| Dimension | General | Technology | Regulated | Security | Commercial | Data | Operations |
|---|---|---|---|---|---|---|---|
| Adoption | 25 | 30 | 15 | 20 | 25 | 25 | 20 |
| Trust & security | 20 | 20 | 25 | 25 | 20 | 20 | 20 |
| Health | 15 | 20 | 10 | 15 | 15 | 20 | 15 |
| Ecosystem | 10 | 15 | 10 | 10 | 15 | 15 | 15 |
| Domain fit | 10 | 10 | 20 | 15 | 15 | 15 | 20 |
| Governance | 5 | 0 | 15 | 10 | 5 | 0 | 5 |
| Docs | 5 | 5 | 5 | 5 | 5 | 5 | 5 |
Weights are normalized by their sum. An unknown component contributes 0 and is flagged; it is never imputed. Applied to the workbook’s own components, these profiles reproduce all 1,200 scores in the 25 Sep 2026 workbook within 0.1 (the import check). Scores are shown as integers; they are comparative discovery signals, not certification.
Evidence tiers
1 Registry-matched (credit 1.0) · 2 Marketplace-listed (0.8) · 3
Document-evidenced (0.7) · 4 Vendor-claimed (0.3) · 5 Unknown (0). Expired,
stale, revoked, in-process and inherited items earn zero and are shown with
their dates. Wording per framework lives in config/frameworks/*.json.
Where evidence comes from (P1b)
| Source | Tier | How it is matched | Cadence |
|---|---|---|---|
FedRAMP Marketplace data.json | 1 | vendor ↔ CSP (legal entity) and product ↔ CSO (offering); accept needs confidence ≥ 0.9 with entity ≥ 0.85 and offering ≥ 0.8, else the pair goes to data/review/matches.yaml | weekly |
| CSA STAR Registry index | 1 | vendor ↔ organization card; the card’s filter terms give STAR Level 1/2, STAR for AI, AIUC-1 and ISO/IEC 42001. Entry pages are not crawled | weekly |
Maintainer-curated (data/curated/compliance.yaml) | 1–4 by what was checked | DPF list, IAF CertSearch, HITRUST letters and any document reviewed by hand; every record carries an evidence URL, a date and who checked | on edit |
Marketplace listings (data/curated/marketplaces.json) | 2 | listing metadata (e.g. AWS “HIPAA eligible”) | weekly |
| Vendor trust/security pages | 4 | robots-aware crawl of at most five pages per publisher domain; a framework named on the page becomes a claim with that page as evidence | weekly, stale after 3 months |
Curated trust pages (data/curated/trust_pages.yaml) | 4 | the certifications page of a large vendor (AWS, Google Cloud, Microsoft, …) that the five-page walk never reaches; read first, then the normal walk. Rows hosted on a shared host (a GitHub repository, an AWS Marketplace listing) are crawled under their vendor’s own domain when vendors: names it, never under the host’s | weekly, stale after 3 months |
Rules that never bend: “HIPAA certified” is never rendered (HIPAA has no
certification; the badge reads BAA available); “GDPR certified” appears
only for a verified Art. 42 seal; a registry match that is only strong on the
vendor name is a review item, not a badge (Oracle SCM Agents is not Oracle
Service Cloud). Records are deduplicated per (framework, variant) keeping the
best tier, then freshness is applied from the framework’s expiry_rule
(fixed_date, period_end_plus_months, recheck_months, mirror_registry)
and a next_check date is set from the source cadence.
Governance from evidence
For each domain a preset of frameworks applies (healthcare: HIPAA, HITRUST,
SOC 2; government: FedRAMP, CSA STAR, ISO 27001; …) plus ISO 42001 and CSA
STAR for every domain. The governance component is the mean over that preset
of the best credited, active record’s tier credit, scaled to 0–100, and
governance_credited on each domain entry lists what counted. Since
sar-score-2.0 this is the only governance figure; there is no separate
workbook value to compare it with.
Trust changelog
Every build diffs each resource’s compliance summary against the last
published index and appends badge_added, badge_changed and
badge_removed events to data/changelog/trust-changelog.jsonl (FR-53).
Changes
Any change to scoring weights, the taxonomy, a framework file or an evidence
rule goes through a reviewed pull request, bumps the relevant version string
and is recorded in data/changelog/trust-changelog.jsonl.
Discovered resources (components-from-signals-1.0)
Resources found by the live connectors have no workbook components, so the seven components are derived from observable signals. Every rule is heuristic and versioned; a component with no supporting signal is unknown (scored 0 and flagged), never imputed.
| Component | Derived from |
|---|---|
| Adoption /30 | log10 of GitHub stars (or Hugging Face likes ×10, downloads /100) against a ceiling of 150,000 |
| Trust /25 | identity tier base (T1 20, T2 18, T3 15, T4 10, T5 6) + 2 per verified protocol, 1 per claimed, +1 signed Agent Card, +1 declared license |
| Health /20 | days since last push or update: <30 → 20, <90 → 16, <180 → 12, <365 → 8, else 4; archived → 2 |
| Ecosystem /15 | MCP verified 6 / claimed 3.6, A2A 5 / 3, ARD 4 / 2.4, +0.3 per topic (max 3), +1 per MCP transport |
| Domain fit /100 | classifier confidence × 100 |
| Governance /100 | tier-weighted credit of active compliance records (Phase 1b); unknown until evidence exists |
| Docs /10 | description 4, homepage 3, representative queries 2, capabilities 1 |
Discovered resources appear in domain lists as unranked entries sorted by score. The seed workbook remains the ranked Top 100 per domain (frozen baseline) until the maintainer re-scores.
What one evidence row says
Every material claim on a dossier is shown as a compact block with the same six fields, so a reader can tell at a glance what is independently supported, what is self-reported and what is unknown:
| Field | Where it comes from | What the reader sees |
|---|---|---|
| Claim | framework + variant | the precise assertion (“SOC 2 Type II”, “FedRAMP Moderate”), never “secure” or “compliant” |
| Subject and scope | scope (entity / product) + covers_resource | whether the evidence is about this agent or about its vendor; vendor-level rows are shown in a separate, muted group and never count toward the agent’s evidence tier |
| Evidence type | tier | T1 registry match, T2 marketplace listing, T3 reviewed document, T4 vendor assertion, T5 nothing found |
| Provenance | issuer, source, evidence_url | who issued it and the page or registry entry it was read from — every row links |
| Timing | issued, retrieved_at, valid_until / period_end / next_check | the as-of date, the expiry where one exists, and when it is re-checked |
| Limitations | the framework’s wording template | what the row does not establish (“covers platform, not this agent”, “a DPA is a contract you still have to sign”) |
Agent-level versus vendor-level: a CSA STAR entry for “Microsoft” is real evidence that the
organisation was assessed and says nothing about whether one product is inside that scope. The
site groups such rows under “About the vendor — inherited” and the trust strip, the compare view
and the homepage specimen count only agent-scoped rows (agentScoped() in web/src/lib/dossier.ts).
FedRAMP has the same split. When no marketplace offering matches the agent but the vendor
holds a FedRAMP authorization for something else, the build adds one vendor-level row for the
vendor’s highest authorized impact level with covers_resource: inherited (tier 1, scope
entity). “Amazon Bedrock Agents” is not on the marketplace; AWS’s authorizations are, and the
dossier shows them in the vendor group, never in the agent-scoped strip.
Before you deploy (deploy-rules-1.0)
Every dossier ends with three lists derived from data, never written per agent:
- Evidence supporting this use — active rows from the ledger (vendor-level ones labelled) and, for agents in the reference set, the deployment questions answered “yes” or “configurable”.
- Unknowns to check — frameworks from the domain preset with nothing found, expired or stale rows, questions the vendor documentation does not answer, and the buyer checklist.
- Suggested restrictions for a pilot — rules in
config/deploy_rules.json, each keyed to a condition (an answer value, a missing framework, an identity tier, a protocol state). They are advice for a bounded pilot, not a verdict.
The reference set (reference-set-1.0)
data/curated/reference_set.yaml holds a small group of agents (two use cases: coding agents in a
private repository; claims-intake agents touching personal data) whose dossiers answer five
deployment questions each from the vendor’s own documentation, with the page quoted and linked.
Values are yes, no, configurable or unknown; unknown means the pages checked do not say
and is a question for the vendor, not a mark against the agent. The file records who collected the
answers and, once a maintainer has re-checked every source, reviewed_by; until then the pages say
“maintainer review pending”. The worked comparisons at /compare/<use-case>/ are rendered from it
and close with one honest sentence per agent, including when the evidence is not sufficient to
scope a pilot. Corrections: the “Dispute or correct this dossier” link on every dossier opens a
pre-filled GitHub issue.
Protocol status (MCP · A2A · ARD)
Each resource carries one status per protocol:
| Status | Meaning | Site glyph |
|---|---|---|
verified | the build fetched and parsed the artefact on the publisher’s own domain: /.well-known/ard.json, the A2A agent card, the MCP server card (or completed an MCP handshake) | ✔ |
claimed | the vendor documents support (data/curated/protocols.yaml, with a reviewer, a date and the documentation URL) or the resource names the protocol in its own tags, description or deployment text; source says which | ○ |
invalid | an artefact was found but did not validate | – |
not_found | the publisher domain was probed and nothing was there. A web page served where the JSON artefact should be (the site’s soft 404) counts as nothing there, not as an invalid artefact | – |
unknown | never probed; not_checked says why (code_host: the URL is on github.com, huggingface.co, pypi.org or npmjs.com, which never carry a publisher’s well-known files; no_publisher_domain) | ? |
Seed rows whose URL is a GitHub repository are looked up once (GET /repos/{owner}/{repo},
agentdossier/connectors/seed_enrich.py) before any probe runs. The repository becomes
external_ids.github (identity tier 3, repository ownership), its homepage becomes the
publisher_domain the probes and the trust-page crawler use (a GitHub Pages or code-host
homepage does not), and its topics join the self-described protocol claims. Without that
step a workbook row such as github.com/crewAIInc/crewAI is “vendor name only” and unknown
on every protocol, however much the project publishes on crewai.com.
A claim never outranks an observation. When a curated claim sits next to a not_found probe the probe result is kept under probe, so the dossier can say “documented by the vendor; no endpoint at the well-known paths”. Protocol trust tier is 1 with any verified, 3 with any claimed, else 5.
Vendor strings that name more than one entity (“GitHub / Microsoft”, “OpenAI (Microsoft)”) are matched against registries part by part and the best part wins, so a product sold under a subsidiary’s name still finds the parent’s FedRAMP and CSA STAR rows.
Identity tiers (identity-1.1)
| Tier | Site wording | Evidence |
|---|---|---|
| 1 | publisher confirmed | ARD manifest with a trustManifest.identity that binds to the publisher domain |
| 2 | publisher confirmed | Marketplace listing; ARD/A2A metadata served at the publisher’s own domain; the resource’s own URL is a page on the vendor’s domain (salesforce.com/agentforce; domains from config/vendor_aliases.json count as the vendor’s); or a publisher domain the maintainer confirmed in data/curated/identity.yaml |
| 3 | publisher likely | Repository or Hub account ownership (GitHub, Hugging Face), or a publisher domain inferred from repository metadata that carries the vendor’s name |
| 4 | publisher unconfirmed | Vendor name only |
| 5 | publisher unknown | — |
Certifications earn governance credit only once identity is tier 2 or better (FR-49). Below that the ledger still shows every row, and the dossier says once, above the ledger, that the publisher is unconfirmed and why; rows are not individually suffixed. The “confirm the publisher” pilot restriction fires at tier 4 only.
Traces
The incident registry (/traces/) is one row per real, public event where an agent attacked or
defended a system — tactics (MITRE ATT&CK), a timeline, what defenders can take from it, and
sources — exportable as JSON and STIX 2.1. It is curated in data/curated/incidents.yaml; see
docs/incidents.md. Alongside it, each dossier carries agent-level trace evidence:
Every dossier lists the public traces of the agent under attack, in defence or in an
incident, with a tier (1 reproducible run · 2 third-party report with transcript · 3
vendor report · 4 vendor claim · 5 none found) and says “none found” when nothing
exists. Sources, the append-only store under data/traces/ and the publication
policy are in docs/traces.md. New traces are produced by a sandbox harness that
runs agents against the AgentDojo prompt-injection benchmark (MIT) and records, per task, whether
the injection succeeded — model-level evidence, never authored attacks.
News and chatter (FR-43–FR-47)
Sources: Hacker News (Algolia, stories with ≥ 5 points), GDELT DOC 2.0 (90 days, English; skipped for the rest of a run after two rate-limit failures), GitHub releases (with a token), the vendor’s own RSS/Atom feed discovered from its homepage, and NVD for CVE counts. Every resource gets its own repository’s releases and its vendor’s feed; only resources with a distinctive name go to the name-search sources (two words, or one non-generic word of six or more letters; “Muse” or “Dify” alone is not searched). A release published by the agent’s own repository links at confidence 1.0 whatever its title; everything else must name the agent in the headline plus one more signal (vendor named, vendor domain, vendor feed).
An item links to a resource only when the resource name appears in the
headline and a second signal agrees (vendor name in the text, publisher
domain in the URL, the vendor’s own feed, or a distinctive multi-word name),
giving confidence ≥ 0.85. Near-identical headlines within 48 hours are
clustered. Ranking is link confidence × source authority (news/security 1.0,
community 0.8, vendor 0.7) × recency (half-life 14 days; 3 for community,
30 for security) × engagement × coverage breadth, with at most three items
per outlet, at most two vendor items and at least two community items when
available. Items tagged security_incident, outage or legal_regulatory
and NVD CVEs feed the resource’s issues block. Only the headline, link,
outlet, date, tag and a 240-character summary are stored (FR-47).
Duplicate handling
Deterministic keys (ARD URN, GitHub owner/repo, Hugging Face id, MCP
Registry name, A2A card URL, canonical URL) merge automatically, with the
seed record winning on descriptive fields. Two workbook products that share
a key stay separate. Name similarity ≥ 0.92 only creates a candidate in
data/review/matches.yaml; nothing merges without a recorded decision.
What the badges on this site mean
| Badge | Meaning | Credit |
|---|---|---|
| T1 | Registry-matched: found on the issuing registry (FedRAMP Marketplace, CSA STAR, DPF list, IAF CertSearch) | 1.0 |
| T2 | Marketplace-listed: a cloud marketplace listing states it (e.g. AWS "HIPAA eligible") | 0.8 |
| T3 | Document-evidenced: the maintainer reviewed a certificate, letter or report | 0.7 |
| T4 | Vendor-claimed: the framework is named on the vendor's own trust or security page | 0.3 |
| T5 | Unknown: nothing found in the sources checked | 0 |
| T1 | Dashed: evidence found but not credited, because the publisher is not yet confirmed (identity worse than tier 2); the dossier says so once above its ledger | 0 |
The trust strip on every result shows five facts, and only one of them is a T-scale: Publisher (confirmed, likely, unconfirmed or unknown — who publishes the agent), Evidence (best active compliance record about the agent itself, T1–T5), Security (an NVD keyword search was run for the score; the dossier shows the CVEs the CVE registry links to the agent, never a count of ours), Protocols (A2A / MCP / ARD observed) and Issues (open incidents, CVEs or regulator actions). "–" means not checked in this build.
How traces and the incident registry are collected
A trace is evidence of what an agent did under attack, in defence or in an incident. It has its own five-step scale, kept apart from compliance tiers: T1 a reproducible run with public artifacts · T2 a third-party report with a transcript · T3 a vendor report · T4 a vendor claim · T5 none found. "None found" is never a finding against the agent — it means nobody has published what the agent does under attack, and the dossier says so.
- The incident registry (/traces/) — one row per real, sourced event where an AI agent attacked, defended or was the target, curated from post-mortems, advisories, researcher disclosures and the press. Every row is "as reported": figures are the sources' own, MITRE ATT&CK tactics are the reviewer's reading, and each row says who reviewed it — a person, or the weekly machine review (marked machine-reviewed). Each has a timeline, lessons for defenders, and STIX 2.1 / JSON export.
- The weekly scout — the Monday refresh reads a curated list of public feeds (press, researcher labs, vendor advisories) and keeps any item that names both an agent and an attack as a candidate: title, link, date, outlet, snippet. A candidate is a lead, never an incident.
- The weekly machine review — publish by default. The same run fetches each candidate's article and asks a model one structured question against that text only: is this a specific, real event in which an AI agent attacked, defended or was the target? The reply (date, role, category, an "as reported" summary, the two sides as the source describes them, tactics, models and agents named, a timeline) is validated field by field; a field the article does not state stays empty. It is published when the verdict is an incident with confidence of at least 0.8 from a source that is not an aggregator; otherwise it is dismissed with the reason kept on the candidate. The model is told never to invent a fact, never to reproduce prompts, payloads or commands, and never to name a private individual. Every decision is listed in the refresh's pull request, which merges itself once the gates pass; a wrong row can be retracted by hand. Nobody waits on a person, and no row is "pending".
- The weekly re-check — every published incident's source links are re-fetched and the result recorded (answers · moved · gone · bot-walled). A link that stops answering is flagged on the incident page; the record itself is never changed by the check.
- Per-agent traces on each dossier — curated write-ups, the registry's own endpoint probes (every refresh records each publisher's ARD/A2A/MCP exchange), and vendor-published bundles declared in an ARD manifest and hash-checked. All live in an append-only, hash-chained store that CI verifies.
- The harness — runs an agent against the AgentDojo prompt-injection benchmark and records a tier-1 bundle. It is built and tested; it produces public traces only when a model-provider key is configured, and until then no public harness trace exists.
The full rules, the review bar and the schema are in docs/incidents.md and docs/traces.md. Behaviour is published, never weapons: scenario inputs and the agent's actions, no payloads against third-party systems.
How the CVE registry is compiled
The CVE list is every record on the official CVE Program's list (via the NVD API 2.0, with CVSS, CWE and CPE as NVD publishes them) that passes one of two tests, rebuilt every Monday since 2023-01-01:
- Test A — it names a listed product. The record's CPE vendor:product, or its exact product name,
matches one of the products in
config/cve_products.json: agent products, agent frameworks, protocol implementations (MCP and A2A servers, clients and SDKs) and the inference and orchestration runtimes agents call through. A product is listed only if it is agentic by design; a general-purpose platform with an AI feature is not, because listing it would list every CVE it has. - Test B — the record is about agents. It carries CWE-1427 (improper neutralization of generative-AI prompts), or its official description uses one strong agent phrase (prompt injection, MCP server, tool calling …) or two distinct weaker ones. Every row keeps the test and the terms that listed it, shown on its page.
Severity, description, affected versions and references are the official record's (NVD's score first, then the CNA's; references NVD tags Exploit are dropped). The category, the attack-surface tags (from the CWE and the description) and the links to agent dossiers and public traces are this registry's classification. CISA's Known Exploited Vulnerabilities catalog is the only exploitation signal carried. A row is a candidate until a person includes or excludes it (its page says who did); a wrong listing is excluded with a one-line, noted override that survives every rebuild, and the record is never edited. No score of ours is derived from a CVE; the dossier's Security row and its CVE band show the registry's links to the agent (by listed product) and stop there — "no CVE linked" reads "none linked in the products checked", never as a finding. A search result shows the same count; an incident page lists the CVE ids its sources name, and a CVE page the incidents and dossiers that name it. Advisories without a CVE id (GitHub-only GHSAs) are not listed. The rules in full: docs/cves.md.