AI Risk Assessment Checklist (AIRAC)

What your team walks away with after completing AIRAC.

• Completed checklist: a signed-off record of controls evaluated and their status.
• Risk register: entries with inherent and residual risk scores, owners, and mitigation plans.
• Decision log: explicit outcomes for each material risk: Accept, Mitigate, Defer, or Stop, with rationale.
• Evidence index: links to artifacts supporting the assessment (tests, benchmarks, design docs, sign-offs).
• Monitoring & incident plan: defined KPIs/alerts, runbooks, escalation paths, and communications.
• Re-assessment dates: scheduled checkpoints (e.g., quarterly or on major change) with owners and triggers.

1. Set context: pick lifecycle stage, risk class, and the feature/use-case ID.
2. Name owners: record R (Responsible) and A (Accountable); confirm risk appetite and phase gate criteria.
3. Work item-by-item: read the check-when statement, attach or link evidence, and tag the evidence level (L1–L4).
4. Run missing tests: schedule any required evaluations (fairness, robustness, privacy, security, abuse).
5. Review: R verifies completeness; A reviews fitness vs. thresholds/tolerances.
6. Decide: mark Accept, Conditional Accept, or Reject; record date and scope of the decision.
7. Publish the pack: export the decision log, evidence index, and risk register; notify co-signers.
8. Plan reassessment: set the next review date, monitoring triggers, and incident/rollback contacts.

• L1: Self-attest (owner statement / policy; no independent check yet)
• L2: Peer-review (reviewed by a qualified peer team; documented feedback addressed)
• L3: Independent internal (independent function inside the org, e.g., QA/Risk/Privacy/Security, with reproducible evidence)
• L4: Independent external (third-party assessment or audit; production-like evidence and formal report)

High-risk items usually require L3+ before release. Record the level for each item.

• High risk: Evidence quality must be L3+ before release.
• Medium risk: Evidence quality must be L2+ before release.
• Low risk: L1 may be acceptable, but L2+ is preferred.

• Responsible (R): does the work and assembles evidence; attests “meets check-when.”
• Accountable (A): owns the risk and accepts or rejects it; must be a named manager.
• Co-signers (when applicable): Legal • Privacy • Security • Safety • Domain owner.

If an item is Conditional Accept, capture all four:

1. Owner: who must complete the condition.
2. Ticket: tracker ID/URL with the exact deliverable(s).
3. Expiry: date/time the condition must be met or re-reviewed.
4. Fallback: throttle/rollback or compensating control if the condition is missed.

Conditional items roll up to the release gate. Any high-risk item that is still conditional at expiry triggers escalation and re-review.

Evidence quality (L1–L4)

• L1: Self-attest: Owner statement or policy. No independent check yet.
• L2: Peer-review: Reviewed by a qualified peer team. Feedback addressed and documented.
• L3: Independent internal: Checked by an independent function (e.g., QA, Risk, Privacy, Security). Reproducible evidence linked to data/code/configs.
• L4: Independent external: Third-party assessment or audit with a formal report and production-like evidence.

Gating: High risk requires L3+ before release. Medium risk requires L2+. (Low risk: L1 may be acceptable, L2 preferred.)

Who signs (R/A)

• Responsible (R): Does the work, assembles evidence, and attests that the check-when criteria are met.
• Accountable (A): Owns the risk and decides Accept / Conditional Accept / Reject. Must be a named manager.

Best practice: one named person per role. Avoid “team” or shared mailboxes for R or A.

When co-sign is required

• Legal: New or high-risk jurisdiction, T&Cs changes, IP/licensing exposures, or material user-rights impact.
• Privacy: Personal data, sensitive categories, cross-border transfers, new purpose, DPIA/FRIA triggers.
• Security: Internet-exposed features, new vendors, model/data exfiltration risks, supply-chain changes.
• Safety: Potential for physical harm, high-stakes decisions, open-ended generation or autonomous actions.
• Domain owner: When domain policy or compliance standards apply (e.g., clinical, financial, public sector).

How to record a Conditional Accept

1. Owner: Person responsible for closing the condition.
2. Ticket: Tracker ID/URL with exact deliverables.
3. Expiry: Date/time for completion or re-review.
4. Fallback: Temporary control (e.g., throttle, feature flag, rollback) if not met by expiry.

Examples

• Accept: "Privacy leakage evaluation L3 independent internal passed across 3 scenarios; evidence: PRJ-42 report; R=Data Science Lead; A=Product Owner; decision date recorded."
• Conditional Accept: "Robustness stress on edge-case set pending; ticket #12345, owner=SecEng, expiry=2025-11-15; fallback=rate limit at 1 rps and human review for flagged cases."

Standardized scales make scoring comparable across teams and releases, and they turn risk into objective go/no-go criteria. Define the scales below with clear examples, then document the numeric thresholds that trigger high-risk status and the decisions each threshold enforces.

Core scales (all systems)

• Likelihood (1–5): shared probability yardstick for how likely a risk scenario is to occur; include concrete examples for each level.
• Impact (1–5): severity of harm across safety, privacy, financial, legal, and reputational dimensions; calibrate “material” impact to appetite and escalation rules.
• Detectability (1–5, optional): how reliably/quickly the issue is noticed in operation; use it if it helps triage and residual-risk judgments, otherwise record why it’s not used.

Use these as starter labels your teams can tailor. Keep the 1–5 levels consistent across programs; swap the examples for your domain.

Likelihood (1–5): "How likely is this to occur in the next release cycle?"

• 1 Very unlikely: <1% chance; requires multiple failures; no prior occurrences in similar launches.
• 2 Unlikely: ~1–5%; rare edge condition; 1 prior instance out of many launches.
• 3 Possible: ~5–20%; plausible with one control failing; a few prior instances.
• 4 Likely: ~20–50%; expected if monitoring misses; several prior instances or near-misses.
• 5 Very likely: >50%; will occur without new controls; repeatedly observed in pilots or analogues.

Impact (1–5): "If it occurs, how bad is it?"

• 1 Minimal: Minor annoyance; no personal data; self-heals; no user or regulatory impact.
• 2 Low: Small cohort affected; reversible with standard support; no legal/contract exposure.
• 3 Moderate: Material inconvenience or minor financial loss; PII involved but contained; complaint risk.
• 4 High: Significant harm to users or business; safety/privacy/legal exposure; regulator contact likely.
• 5 Severe/Catastrophic: Physical harm or major rights impact; large financial/regulatory exposure; immediate rollback and notify.

Detectability (1–5, optional): "How reliably/quickly will we notice?"

• 1 Easy: Automated alerts fire before users notice; precise signals; <1 hour to detect.
• 2 Good: Alerting covers most scenarios; same day detection.
• 3 Limited: Signals noisy or lagging; detection within a few days.
• 4 Poor: Hard to spot without manual review; detection likely weeks later.
• 5 None: No realistic detection until after external reports/incident; post-hoc discovery only.

GenAI-specific scales

• Groundedness / Confidence: e.g., grounded / partial / ungrounded for generated outputs; tie to factuality checks and user disclosures.
• Exposure / Reach: # users affected or outputs produced; higher reach warrants stricter thresholds and faster remediation.
• Reversibility / Velocity of harm: how fast harm occurs and how reversible it is; rapid/irreversible harms justify conservative releases and readily available kill-switches.
• Human-oversight coverage: who reviews, when, and how; quantify coverage to validate oversight design and staffing.

High-risk thresholds & decision rules

Document the numeric rules (by phase and use-case type) that flip an item to high-risk and link gates to those rules. Example patterns include: Likelihood ≥4 and Impact ≥3; or composite score ≥ a defined cutoff. These rules must be referenced by release gates and escalation paths.

How gates work

• Design/Testing gates: if a threshold is met or exceeded, add controls or strengthen evidence (e.g., move from L2 to L3) before proceeding.
• Pre-release gates: "high-risk" status triggers mandatory escalation and stronger oversight; the gate decision is Accept / Conditional Accept / Reject with rationale.
• Post-release gates: monitoring/incident thresholds (e.g., drift, safety, abuse) force reassessment or rollback per the oversight and kill-switch plan.

Assumptions & uncertainty

List major assumptions and known unknowns; tie each to a follow-up action. Making uncertainty explicit reduces overconfidence and directs extra evaluation where risk concentrates.

Notes: AIRAC is a publicly available checklist with an evidence-quality model and release gates. Other entries are regulator/standards frameworks or public-sector tools. Where they specify controls but not an L1–L4 scheme, we describe how evidence is typically produced (e.g., test artifacts, audits). Links go to official publishers.

Callouts for GenAI deployments and the corresponding controls to apply with AIRAC.

• Prompt governance: version prompts, review/approve changes, protect secrets.

  Why this matters: Prompts steer behavior; unmanaged edits can leak secrets or regress safety.

• Sampling configuration: set temperature/top-p/seed, max tokens, and stop sequences per use case.

  Why this matters: Tuning controls variance and determinism; mis-set values cause instability and risk.

• RAG freshness & TTL: define index recency, cache TTLs, and invalidation on source updates.

  Why this matters: Stale context produces wrong answers; TTLs enforce timely refreshes.

• Injection & jailbreak testing: red-team prompts, canaries, and end-to-end evals pre-release.

  Why this matters: Adversarial prompts can bypass guardrails; testing reduces abuse paths.

• Toxicity & PII benchmarks: measure harmful content and leakage with representative data.

  Why this matters: Quantifies safety posture and verifies mitigations before scale.

• Watermark & provenance: mark AI-generated content and record chain-of-custody.

  Why this matters: Traceability enables disclosure, detection, and downstream trust.

• Tool permissioning: least-privilege scopes, explicit approvals, and auditable logs.

  Why this matters: Model-invoked tools can cause real-world changes; restrict scope and audit.

• High-stakes human gating: mandatory human review for safety, legal, or financial outcomes.

  Why this matters: Human oversight reduces catastrophic error in high-impact decisions.

AIRAC is licensed under Creative Commons Attribution–NonCommercial 4.0 International (CC BY-NC 4.0) . You may share and adapt the checklist for non-commercial purposes with proper attribution.

What you may do

• Share: copy and redistribute in any medium or format.
• Adapt: remix, transform, and build upon the material.

What you must do

• Attribution (TASL): Include Title, Author, Source (link), and License (link); indicate if changes were made.
• NonCommercial: Use must not be primarily intended for commercial advantage or monetary compensation.
• No endorsement: Don’t imply the licensor endorses you or your use.
• No additional restrictions: Don’t add legal/technical terms that restrict others from doing what the license permits.

How to attribute (TASL example)

AI Risk Assessment Checklist (AIRAC v0.04, 2025) by Richard R. Khan — RAISEF, https://raisef.ai/tools/airac, licensed CC BY-NC 4.0; changes: [describe].

Logos & third-party material

"RAISEF" names and logos are not covered by the CC license. Any third-party content keeps its own rights; mark and respect those licenses.

Commercial use

For uses outside “NonCommercial” (e.g., resale, paid training bundles, inclusion in commercial products), request a separate license: licensing@raisef.ai.

Good practice

• Link to the license and note changes made.
• For web/PDF, embed license metadata (e.g., HTML rel="license" or PDF/XMP) so attribution is human- and machine-readable.

Case Study 1: AI-Driven Healthcare Diagnostics

Case Study 2: Fairness in AI-Driven Credit Scoring

• More

Join the RAISEF community or request a briefing. Share a few lines about your goals and timelines; we’ll follow up soon.

• support@raisef.ai
• Toronto, Ontario
  CANADA

We don’t collect assessment data through this site. If your note includes sensitive information, send a brief summary and we’ll arrange a secure channel.