Testing and Testability

Definition

Testing and Testability

Testing checks whether software behaves as intended under known conditions, while testability is the degree to which a system can be controlled, observed and isolated during those checks. AI systems need conventional unit, integration and end-to-end tests for deterministic code, plus evals and error analysis for probabilistic model behaviour, retrieval quality and tool-use workflows.

Why It Matters

The testing pyramid combines unit, integration, contract, system and acceptance tests. Traditional tests verify deterministic behaviour; evals verify probabilistic behaviour. Both are needed in AI applications.

Testability means designing components so dependencies, data and model calls can be controlled or substituted. It makes failures reproducible and lowers evaluation cost.

Failure mode

The recurring risk is treating this concept as a label rather than an engineering decision with trade-offs, evidence needs and operational consequences.

Evaluation check

Ask what observable evidence would show that this concept has been applied well in a real system, repository or learning exercise.

Related Concepts

• Evals and error analysis
• Evaluating coding agents

Further study: Kent Beck, Test Driven Development, and Martin Fowler, Testing.