AI Incident Response Plan

Standard: ISO/IEC 42001:2023 Annex A.9.4 Document owner: Platform Engineering Lead Review cadence: Quarterly; after any P1/P2 incident Classification: Internal — Restricted

1. Severity Levels

2. Incident Scenarios and Response Procedures

2.1 Gateway Complete Unavailability (P1)

Detection: Health probe /v1/health returns 5xx; alert fires.

Response:

1. On-call engineer acknowledges alert within 15 min
2. Check pod logs: kubectl logs -n agentrust-edge deploy/gateway --tail=200
3. Check database connectivity: verify PostgreSQL pod is running
4. Check Redis connectivity: redis-cli -a $REDIS_PASSWORD ping
5. If DB migration failed on startup: rollback deployment, investigate migration
6. Escalate to Platform Lead if not resolved in 30 min
7. Post status update to internal incident channel every 30 min

Recovery: Roll back to previous deployment image tag; re-run smoke tests.

2.2 Systematic Harmful Approvals — False Negatives (P1)

Detection: Approval rate exceeds normal range; human reviewer reports pattern; external report via /v1/feedback/adverse-impact.

Response:

1. Immediately set decision thresholds to block all high and critical risk tier — pause auto-approve
2. Pull recent decisions: GET /v1/audit/executions?decision=approve&limit=100
3. Identify common attributes (agent_id, framework, policy pack version)
4. If policy pack issue: disable affected pack, revert to previous version
5. Notify affected operators via email within 2 hours
6. Conduct manual review of past 24h of approved executions for harm

Recovery: Re-enable approval only after root cause is identified and policy corrected; re-test against adversarial golden test suite.

2.3 LLM Judge Engine Failure (P2)

Detection: Circuit breaker opens (_circuit["open"] == True); alert judge_failure_rate fires.

Immediate action: Judge returns None — governance pipeline continues without judge score (safe degradation). Decision engine uses available signals only.

Response:

1. Check circuit breaker state: inspect logs for "Judge circuit breaker is open"
2. Test judge backend manually: curl http://localhost:11434/api/chat (Ollama) or Anthropic API test
3. If Ollama: check pod resources, model download status
4. If Claude API: check ANTHROPIC_API_KEY validity; check Anthropic status page
5. Reset circuit breaker by restarting the gateway worker (the breaker resets after 60s automatically)

Recovery: Confirm judge scores are being generated again; verify sampling rate is correct.

2.4 Audit Ledger Hash Chain Corruption (P1)

Detection: GET /v1/audit/verify-chain returns integrity failure; alert ledger_integrity_failure fires.

Response:

1. Immediately halt all writes to the audit ledger (set AUDIT_READONLY=true env var and restart)
2. Identify the first corrupted record by bisecting the chain
3. Do NOT delete records — preserve all evidence
4. Restore from last known-good backup
5. Replay executions that occurred after the last backup if possible
6. Notify affected operators; document in incident log

Recovery: Verify chain integrity passes before re-enabling writes.

2.5 Approval Rate Anomaly (P2)

Detection: Alert approval_rate_drop fires (>20% change in rolling 1h window).

Response:

1. Pull analytics: GET /v1/analytics/summary?window=60
2. Check if a policy pack or threshold config was recently changed (git log config/)
3. Check for increased adversarial input traffic (grep adversarial gateway.log)
4. If config change: roll back to previous version
5. If traffic anomaly: check rate limiter effectiveness; increase sensitivity if needed

2.6 Data Breach — PII Exposed in Logs or External Systems (P1)

Detection: PII detection alert fires; manual discovery; external report.

Response:

1. Identify scope: which envelopes contain exposed PII? (grep -i "ssn\|password\|api.key" audit_records)
2. Rotate affected credentials immediately
3. Notify Data Protection Officer within 1 hour
4. If GDPR breach: notify supervisory authority within 72 hours of discovery
5. Notify affected data subjects if high risk (Article 34 GDPR)
6. Redact PII from audit records using data subject deletion API

2.7 Authentication Bypass (P1)

Detection: Anomalous traffic with non-JWT tokens; unusual tier-level access patterns.

Response:

1. Rotate AGENTRUST_JWT_SECRET immediately (all existing tokens invalidated)
2. Pull access logs for the past 24h; identify all requests that bypassed auth
3. Invalidate all active API keys via AGENTRUST_API_KEYS rotation
4. Conduct forensic review of audit ledger for unauthorised executions
5. Notify affected customers

3. Post-Incident Review

Every P1 and P2 incident requires a written post-mortem within 5 business days:

Template:

Incident ID: INC-YYYY-NNN
Date: 
Severity: 
Duration:
Impact:

Timeline:
- [TIME] Alert fired
- [TIME] Engineer acknowledged
- [TIME] Root cause identified
- [TIME] Mitigation applied
- [TIME] Resolved

Root Cause: 

Contributing Factors:

Corrective Actions:
1. [ACTION] — Owner: [NAME] — Due: [DATE]
2. ...

Lessons Learned:

Post-mortems are reviewed in the monthly AI Governance Committee meeting and feed into the risk register update.

4. Contact List

5. Document History