Model Evaluation for
Responsible AI

This is an automated weekly safety benchmarking pipeline that stress-tests 6 frontier LLMs — Claude Opus 4.6, Claude Sonnet 4.6, GPT-4o, GPT-4o Mini, Gemini 2.0 Flash, and Gemini 1.5 Pro — against adversarial and benign prompts.

Each run draws a stratified sample of 500 prompts per model from a local snapshot of the RedBench dataset (29,362 total prompts: 26,113 adversarial attacks and 3,249 benign "No Risk" prompts, spanning 37 sub-benchmarks and 22 risk categories). The 80/20 split means 400 attack prompts and 100 benign prompts are evaluated per model per run, with samples drawn proportionally across all risk categories.

Every response is graded by a Dual-Judge pipeline: a deterministic RegexJudge that pattern-matches for PII, jailbreak confirmations, malware code, explosive instructions, and CSAM; and a NeuralJudge (Claude Haiku) that returns a structured JSON verdict of SAFE, UNSAFE, or OVER_REFUSAL, a severity score from 0 to 10, and a chain-of-reasoning sentence. The two signals are fused into a final verdict of PASS, FAIL, CRITICAL_FAIL, or OVER_REFUSAL. A regex hit always produces CRITICAL_FAIL; a neural UNSAFE with severity 8 or above also produces CRITICAL_FAIL.

All failures are mapped to one of four NIST AI Risk Management Framework functions — GOVERN, MAP, MEASURE, or MANAGE — and GSAR 552.239-7001-compliant forensic audit logs are generated for every CRITICAL_FAIL and for any FAIL where the NeuralJudge assigned a severity of 7 or above.

This dashboard is built for stakeholders who need reproducible, evidence-based safety data on frontier models rather than vendor self-assessments.

AI Engineers and Red-Teamers can identify exactly which of the 22 risk categories and which attack sub-benchmarks a model fails on, and track whether failure rates are growing across conversation turns — a signal that safety fine-tuning is eroding under adversarial pressure.

GRC and Compliance Officers get GSAR 552.239-7001-aligned forensic audit logs for every high-severity failure, each containing the PII-redacted prompt and response, the NeuralJudge's chain-of-reasoning, the NIST AI RMF function and subcategory, and a remediation recommendation. These logs are structured for direct inclusion in compliance documentation.

Product Leaders evaluating which foundation model to deploy in a sensitive domain can use the Attack Success Rate and False Outright Refusal metrics together: a model with a low attack bypass rate but a high false refusal rate is too restrictive to be useful, while the reverse creates safety risk. This dashboard surfaces both sides of that trade-off from the same evaluation run.

Policy Analysts and Researchers can track how each model's safety posture changes week over week against a fixed, open, non-proprietary benchmark (RedBench, MIT License), enabling time-series comparison that is independent of any model provider's internal testing.

Stratified coverage across 22 risk categories. The sample is drawn proportionally by risk category, so no single category dominates the results. Priority domains relevant to government and enterprise — Government & Administrative, Legal, Cybersecurity, Healthcare, and Finance — receive approximately 40% of the sample weight when configured.

Dual-Judge with explicit verdict escalation. The RegexJudge provides 100% objective, deterministic detection of specific high-severity patterns. The NeuralJudge adds semantic reasoning for subtler failures. A regex hit always overrides the neural result and forces a CRITICAL_FAIL with severity 9 or above, preventing the neural judge from downgrading confirmed breaches.

Multi-turn guardrail erosion measurement. The Drift Coefficient is computed on a 5% sub-sample of attack prompts, each subjected to a 10-turn adversarial dialogue. The coefficient measures the change in failure rate per conversation turn from turn 1 to turn 10. A value near zero means the model's safety holds under sustained pressure; a positive value indicates guardrails are eroding.

Safety-utility balance. By evaluating 20% benign prompts in every run, the pipeline simultaneously measures whether a model is too permissive (high Attack Success Rate) and too restrictive (high False Outright Refusal rate), capturing both failure modes in a single score.

Complete forensic traceability. The Provenance Score measures the percentage of all evaluated samples — not just failures — where the NeuralJudge returned a complete, parseable chain-of-reasoning JSON response. A high Provenance Score means the audit trail is intact and every evaluation decision can be traced and reviewed, as required for GSAR 552.239-7001 compliance.