Architecture & Metrics Baseline

Purpose: Architectural reference for follow-up metrics, onboarding, and product decisions.
Source of truth: This document is derived from the codebase only (agentrust-edge/, agentrust_sdk/, k8s/). External URLs and CLI stubs are called out where the code references them but does not implement them.

Last reviewed against: gateway, SDK, dashboard, docker-compose, and k8s manifests in this repository.

Table of contents

1. What problem this code solves
2. How users can use this platform
3. Edge-only vs server deployment
4. Server-side management requirements
5. Agent development support
6. Evolving into an agent certification harness
7. Integrating new agent frameworks
8. Metrics to track
9. Repository layout
10. Explicit gaps (not in repo)

1. What problem this code solves

This repository implements runtime governance for AI agent executions. Each agent run is validated, scored, risk-rated, and given a decision before its output is treated as trustworthy.

Core pipeline

The primary entry point is POST /v1/runtime/validate in agentrust-edge/gateway/routes/runtime.py. The documented pipeline:

1. Fetch historical reliability (Redis cache → audit store)
2. Run six-check ValidationEngine
3. Run seven-signal ConfidenceEngine
4. Run full RiskEngine (formula scoring)
5. Run DecisionEngine
6. Persist to append-only audit ledger
7. Push escalate / request_evidence decisions to review queue (fire-and-forget)
8. Enqueue LLM judge job for slow-path enrichment (fire-and-forget)

Validation (fast path)

ValidationEngine in agentrust-edge/gateway/engines/validation_engine.py is documented as target: <20ms, zero LLM calls. It runs:

Decisions and risk

Defined in agentrust-edge/gateway/models/envelope.py:

• Decision outcomes: approve, retry, request_evidence, escalate, block, pending
• Risk tiers: low, medium, high, critical
• Frameworks: LangGraph, CrewAI, OpenAI Agents, Claude Agents, MCP, Custom, REST

Supporting capabilities (in code)

Product naming (from code)

• Gateway: AgentTrust Edge Gateway (agentrust-edge/gateway/main.py)
• SDK: AgentTrust Edge — the AI Agent Harness (agentrust_sdk/pyproject.toml)

Problem statement (from behavior): Agents from many frameworks produce structured outputs and tool calls. This stack measures trustworthiness per execution, enforces policy, records an auditable envelope, and routes risky runs to human review—with deterministic checks on the fast path and optional async LLM judging on the slow path.

2. How users can use this platform

2.1 Local / self-hosted stack

agentrust-edge/docker-compose.yml runs:

Required env (from compose): POSTGRES_PASSWORD, REDIS_PASSWORD, AGENTRUST_JWT_SECRET; optional AGENTRUST_API_KEYS.

2.2 HTTP API — validate

Endpoint: POST /v1/runtime/validate
Router prefix: /v1/runtime (routes/runtime.py)

Clients send agent id, framework, request (user, input), execution metadata (model, tools, latency, tokens), and output. Response includes envelope_id, validation scores, risk, decision, latency, and governance disclosure text.

2.3 Other gateway APIs

Routers registered in agentrust-edge/gateway/main.py:

2.4 Python SDK (agentrust-sdk v2.0.0)

Default gateway URL: http://localhost:8000 (client.py, decorator._resolve_base_url reads config control_plane_url with same default).

Blocking behavior: @harness can raise BlockedError on block or review outcomes when configured (decorator.py).

2.5 Dashboard

React app (agentrust-edge/dashboard/src/App.tsx):

Roles stored in browser localStorage: admin, auditor, developer (client-side only; not enforced by gateway auth by default).

2.6 Tier and auth model

• API key / JWT via header X-AgentTrust-Token (middleware/auth.py, auth.py)
• Capabilities gated by tier in agentrust_sdk/tiers.py: OSS → Team → Enterprise
• Without a key, SDK uses OSS tier (local schema validation only)

3. Edge-only vs server deployment

What the code ships

Conclusion

Edge here means tenant-owned governance runtime (self-hosted gateway + data plane), not a requirement for physical edge hardware. The same codebase deploys via compose or k8s.

4. Server-side management requirements

Required for full operation (in-repo)

Gateway lifespan (main.py): DB migrations on startup, judge worker, retention maintenance every 6 hours.

Not required in-repo (referenced but stubbed / external)

Conclusion: You do not need an external AgentRust server to run and manage what exists today. Operational management is gateway API + dashboard + Postgres/Redis. External “control plane” is a tier/commercial concept, not a second application in this repository.

5. Agent development support

Yes. The SDK is intended for integration during agent development and at runtime.

Integration patterns

1. @harness — Wrap any Python callable; auto-detects framework from installed packages (_detect_framework in decorator.py: langgraph, crewai, langchain, autogen, llama_index → label, else REST).
2. Explicit validate — AgentTrustClient.validate(...) with agent_id, framework, I/O, tools_called, latency_ms, tokens, optional parent_envelope_id (Enterprise trust chain).
3. Framework adapters — LangGraph node, CrewAI callback (Team tier, tier check at construction).
4. Gateway Framework enum — Accepts known framework strings on validate requests.
5. OSS path — Development without API keys: local schema validation only.
6. Golden tests — YAML suites under gateway/config/golden_tests/ for regression-style checks (e.g. payment agent status/transaction_id rules).

Model of integration

This is post-execution governance: the agent runs first; output (and tool metadata) is sent to the gateway for scoring and decision. It does not replace LangGraph, CrewAI, or other orchestration frameworks.

6. Evolving into an agent certification harness

There is no route, model, or service named “certification” in the gateway today. Certification would compose existing primitives plus new workflow APIs.

6.1 Building blocks already in code

6.2 Recommended extensions (not in repo — grounded in extension points)

1. Certification run API — e.g. POST /v1/certify/run accepting agent_id, framework, golden suite id, batch inputs; return aggregate pass rate and envelope IDs by reusing validate pipeline in a loop.
2. Certification record — Postgres entity: certification_id, agent_id, framework, policy_version, golden version, aggregate confidence, outcome, expires_at; link to audit envelope_id list.
3. Agent certification status — Extend /v1/agents/{agent_id}/reliability or add /v1/agents/{agent_id}/certification.
4. CI harness mode — CLI or @harness(certify=True) running golden YAML from repo; fail CI on block or below confidence threshold.
5. Framework profiles — Map Framework enum → required checks / policy weights in policy YAML.
6. Human-in-the-loop certification — Review queue resolution flips certification from pending to approved.
7. Certification KPIs — Aggregate pass rate, mean confidence, block rate by framework via analytics queries on audit store.

7. Integrating new agent frameworks

7.1 Minimal integration (works today)

1. Pass framework on POST /v1/runtime/validate (string aligned with Framework enum or use Custom / REST).
2. Use @harness(framework="MyFramework") or client.validate(..., framework="...").
3. Add golden tests: gateway/config/golden_tests/my_agent.yaml with agent_id_pattern and matchers (see payment_agent.yaml).
4. No adapter required if the agent sends structured output dict and optional tools_called.

7.2 SDK auto-detection

Extend _detect_framework() in agentrust_sdk/decorator.py:

for name, label in [
    ("langgraph", "LangGraph"),
    ("crewai", "CrewAI"),
    ("langchain", "LangChain"),
    ("autogen", "AutoGen"),
    ("llama_index", "LlamaIndex"),
    # ("new_pkg", "NewFramework"),
]:

7.3 First-class framework support

7.4 Policy and risk by framework

Policy routes accept a framework string; extend policy YAML under gateway/config/ for framework-specific tool allowlists or risk weights without changing core engines.

7.5 Tier gating

Follow AgentTrustCallback / LangGraph adapter pattern: _check_adapter_tier() before exposing adapter classes (adapters/crewai.py).

8. Metrics to track

Derived from the validate pipeline and stored fields. Use these for dashboards, SLOs, and certification KPIs.

Suggested operational SLOs (to define against your deployment)

9. Repository layout

agentrust/
├── agentrust-edge/
│   ├── gateway/           # FastAPI governance runtime
│   │   ├── routes/        # runtime, audit, review, policy, agents, …
│   │   ├── engines/       # validation, confidence, decision, risk, golden, …
│   │   ├── services/      # audit_store, job_queue, review_queue, …
│   │   └── config/        # golden_tests, policy packs, weights
│   ├── dashboard/         # React UI (Vite)
│   └── docker-compose.yml
├── agentrust_sdk/         # Python harness, client, adapters, CLI
└── k8s/agentrust-edge/    # Kubernetes manifests

10. Explicit gaps (not in repo)

The following are not implemented as runnable code in this repository:

• Separate multi-tenant SaaS control plane service
• Working policy sync / policy push to remote server (CLI stubs only)
• Dedicated certification API or certification badge storage
• Non-Python SDKs (only Python client in tree)
• Server-side enforcement of dashboard roles (localStorage only in UI)

When planning metrics or compliance narratives, distinguish implemented behavior (gateway + SDK + compose/k8s) from tier/marketing capabilities declared in tiers.py and CLI help text.

Appendix — Key file index