Trustworthy Agency and Evaluation

Definition

Trustworthy Agency and Evaluation

Trustworthy agency combines bounded autonomy, epistemic uncertainty, privacy, safety, alignment, governance and longitudinal evaluation so that agent development remains useful, corrigible and auditable.

Why It Matters

Long-lived social agents introduce risks that single-turn benchmarks miss: stale memories, identity drift, dependency formation, hidden persuasion, privacy erosion, boundary drift, unwanted initiative and relationship manipulation.

Normative architecture represents and enforces behavioral norms. It includes constitutional rules, role norms, user preferences and situational policies. These layers can conflict, so the harness needs explicit mediation, approval gates and audit trails.

Autonomy should be budgeted. Initiative, scope, consequence, reversibility, social sensitivity and temporal span all affect whether an agent may act without fresh approval. More autonomy is not automatically better; it is justified only where the user can understand, revoke and audit it.

Evaluation must be multi-timescale. It should cover task competence, memory quality, relational quality, governance, development and robustness across sessions, projects and longer relationships.

Engineering Mode

Design evaluation suites that simulate weeks or months of interaction. Include memory correction, model migration, emotionally ambiguous prompts, autonomy revocation, role-boundary pressure, stale preferences and adversarial attempts to exploit relational memory.

Evaluation check

Ask whether the system can show evidence of safe development, not only good single-turn answers.

Real Example

A high-trust agent should not contact third parties, reshape a relationship norm, spend money or make identity-relevant claims without stronger authorization, even if prior interactions suggest the user often accepts initiative.

Related Concepts

• Human oversight and guardrails
• Evals and error analysis
• Co-adaptation and Trust

Derived from the source document’s trustworthy-agency research challenges, including memory governance, identity continuity, autonomy control, relational safety and longitudinal evaluation.