A test is evidence for a decision, not a ritual. This page sets out what four practitioners
teach about testing, how we settle the places where they pull in different directions, 160 standards to adopt
from, a release gate, and what changes when an AI writes the code or is the code.
Labels matter here. Paragraphs that describe an author's view say so and cite a source
(S1 to S33, listed at the end). Paragraphs marked My interpretation or synthesis
are our own readings and adoption rules, not claims the authors made or agree on.
1. Four views
Kent Beck: confidence is an outcome, not a ritual
Beck's Test Desiderata describes twelve qualities: isolation, composability, determinism, speed, writability,
readability, sensitivity to behavior, insensitivity to structure, automation, specificity, predictive usefulness,
and confidence. These qualities can conflict; the goal is a useful balance. A realistic integrated check may be
slower but reveal a failure a tiny test cannot. S1
His later account of TDD names ease, reliability, sustainability, and responsibility as desired outcomes. He also
specifies prerequisites: reasonably predictable results and cases, useful composition from small checks into larger
confidence, and manageable test costs. He explicitly permits a different workflow when those conditions fail. S2
My interpretation: Preserve the feedback mechanism and the ability to change code
safely. Do not mistake adherence to the sequence for evidence that the whole product works.
Michael Feathers: discover what exists before deciding what to preserve
Feathers describes characterization testing as documenting actual behavior. An initially guessed expectation can
be replaced with the observed result, because the immediate goal is learning and detecting later changes. That is
different from establishing that the behavior meets a requirement. He recounts users depending on behavior a
developer considered a bug. S3
His seams concept identifies substitution points that let a test control troublesome dependencies. His
discussion of sprouting describes adding independently testable behavior beside difficult existing code. S4
My interpretation: Separate three questions: What happens now? What must remain
compatible? What should change? A characterization test answers the first and helps with the second; it does not
decide the third.
Feathers also warns that AI-generated tests can approve unintended behavior in AI-generated implementation. His
proposed response is to express constraints as executable tests before generation. Passing those tests still
depends on whether the constraints are adequate. S31
Gerard Meszaros: tests have architecture and maintenance costs
Meszaros treats difficult test code as a potential productivity bottleneck. His worked refactoring example
progressively exposes intent and simplifies the test rather than accepting a complicated fixture and verification
block as inevitable. His book organizes both test patterns and the smells that indicate trouble. S5 S6
His chapter on doubles is explicitly conditional: substitutes help when dependencies are unavailable,
uncontrollable, slow, or produce unwanted effects. But he warns against replacing the behavior being verified and
recommends checking important behavior without a double. He also identifies excessive specification through
doubles as a source of fragile tests. S7
My interpretation: A test suite is a product used by engineers. Its users need
understandable failures, reliable execution, and affordable adaptation. Treat brittleness as a defect in that
product.
Glenford Myers: the tester must resist reassuring answers
Myers's psychology chapter challenges definitions of testing centered on demonstrating success or absence of
errors. His principles chapter explains why an expected result must be specified: without one, a plausible output
can be accepted simply because the observer wants success. S8 S9
My interpretation: Approach a feature with a concrete theory of how it could be
wrong. A test discovering a real problem has delivered value. This complements Beck: challenge the system to earn
confidence rather than treating confidence as the starting assumption.
The accessible book previews are partial. This profile does not pretend to reproduce or newly verify
every guideline in the complete book. Later editions have coauthors; the source register identifies the edition used.
2. Resolving the apparent disagreements
These are recommended decisions, not claims that all four authors agree.
Tension
Recommended resolution
Confidence versus error discovery
Seek disconfirming evidence, then calibrate confidence to what survived.
Correctness tests versus characterization
Label intended behavior and observed legacy behavior separately.
Test first versus test after
Prefer test first where outcomes are knowable and costs reasonable; obtain other evidence when they are not.
Isolation versus realism
Isolate for control and diagnosis; integrate to test boundary assumptions.
Fast tests versus realistic environments
Keep development feedback fast and use scheduled or release-stage checks for expensive risks.
Public API versus internal seam
Prefer stable behavior boundaries, but introduce a seam when it makes otherwise inaccessible risks testable.
One assertion versus many
One coherent purpose matters more than an assertion count.
Mocks versus real collaborators
Use a substitute to control a dependency; test the relevant real contract elsewhere.
Coverage versus confidence
Coverage exposes omissions in execution. It does not establish the quality of the oracle.
Determinism versus randomness
Generate varied cases while retaining reproducible inputs and traces; separately characterize unavoidable variability.
Automation versus exploration
Automate repeatable checks and preserve human investigation for new risks and questions.
Universal rules versus economics
Make the evidence proportional to the consequence, uncertainty, and cost of failure.
Fowler distinguishes classical and mockist approaches rather than presenting one universal arrangement. Rapid
Software Testing emphasizes exploration and critical evaluation beyond automated checking. These are useful
counterweights to a purely mechanical doctrine. S10 S32
3. How to use the standards
Select the sections relevant to a component or system. For each applicable rule, record covered,
gap, or accepted exception, with evidence and an owner. Do not score an inapplicable hardware
requirement against a simple web utility.
Treat the wording as an adoption starting point. If a domain has legally required or safety-specific standards,
those need a separate applicability assessment; this reference does not certify conformance.
Foundations: S2, S8, S9, S11, S12. These are synthesized adoption rules.
Define the decision a test informs. A result should help someone accept, reject, investigate, or change something.
Rank testing by failure impact, exposure, and uncertainty. A rare catastrophic failure may deserve more attention than a frequent cosmetic defect.
Write down critical invariants: conditions that must hold across operations, failures, retries, and recovery.
Separate verification of requirements from validation of user needs. Correct implementation of the wrong requirement is still a product failure.
State what passing actually establishes, including the tested inputs, environment, assumptions, and observation window.
Treat passing tests as bounded evidence. Do not present a finite suite as proof of unrestricted correctness.
Challenge the specification itself. Conflicting examples, ambiguous units, and impossible promises are defects worth discovering before implementation.
Use diverse evidence where consequences justify it: tests, review, static analysis, models, operational observations, and user evaluation.
Define release criteria before seeing results; record exceptions with an owner, rationale, and resolution date.
Scale rigor to the system. Apply relevant standards deliberately rather than imposing every technique on every small change.
B. Expected results and test oracles
Foundations: S8, S9, S18, S19. An oracle is the basis for judging a result.
Establish expected behavior independently of the implementation whenever testing correctness.
Do not compute expected values by calling the same production algorithm being checked.
Use independently calculated examples for business rules, especially money, rounding, units, thresholds, and dates.
Assert meaningful results and state changes. A successful status code alone rarely establishes business success.
Check prohibited side effects: no unauthorized write, duplicate delivery, unintended charge, or data disclosure.
Distinguish observed historical behavior from approved intended behavior in test names or metadata.
Use explicit tolerances for approximate results, justified by the domain and measurement uncertainty.
Use metamorphic relationships when exact answers are difficult: appropriate transformations should preserve or predictably change results.
Use differential testing against another implementation where valuable, while recognizing shared bugs and disagreements need independent adjudication.
Review large snapshots as specifications. Updating a baseline requires an explanation of why the changed behavior is acceptable.
C. Readable and diagnostic tests
Foundations: S1, S5, S6.
Name the scenario and outcome so failures explain the broken promise.
Make setup, action, verification, and cleanup distinguishable.
Give each test one coherent behavioral purpose; use several assertions when they jointly express that purpose.
Review changes to tests and expectations with the same seriousness as implementation changes.
For AI-written code, derive acceptance criteria from requirements before allowing generated implementation to define success.
Do not let the same generation process be the sole judge of both code and its correctness.
Require explicit review before an agent weakens assertions, updates baselines, removes tests, or changes evaluation criteria.
For nondeterministic AI systems, use representative held-out cases, repeated trials, task-level scoring, and explicit uncertainty.
Evaluate agent actions and external effects as well as answers, including tool failures, unauthorized actions, cancellation, and bounded resource use.
5. A practical release gate
This gate is my synthesis; tighten or simplify it for the risk.
The changed behavior and important prohibited outcomes are explicit.
At least one meaningful check can distinguish the intended behavior from a plausible faulty implementation.
Relevant component tests and boundary checks have run on the identified artifact.
Critical workflows have appropriate integrated evidence.
Data, security, concurrency, performance, and recovery risks were assessed; relevant checks ran.
Failures, flakes, skipped tests, and untested assumptions are visible.
Expected-result changes have an explanation independent of the new implementation.
Deployment and recovery are feasible for the current schema and external effects.
The residual uncertainty is stated, and a responsible owner accepts any material exception.
A trivial text adjustment should not trigger a disaster-recovery exercise. A change to charge submission, tenant
authorization, or irreversible data transformation should receive much stronger evidence.
6. Worked applications
Payment submission
An illustrative scenario for a Rails service using an external gateway, not a claim about the current
behavior or capabilities of a specific integration.
Requirement: An authorized payment attempt uses the correct merchant, amount, currency, and
invoice; retries do not create prohibited duplicate financial effects.
Risk
Valuable test
Necessary observation
Wrong merchant or invoice
Cross-tenant and mismatched-identifier submissions
Denial before external submission; no local mutation
Client changes amount
Altered client-supplied amount
Server-authoritative amount or explicit validation
Two simultaneous submissions
Synchronize contenders at the critical boundary
External effects and local records remain within contract
Gateway accepts but response is lost
Simulate an accepted request followed by transport failure
Unknown outcome is represented; no blind duplicate submission
Process dies after charge
Crash before local success persistence
Reconciliation reaches the correct state without prohibited replay
Duplicate or reordered webhook
Replay event IDs and reorder event histories
No duplicate effect or invalid state regression
Stale payment status
Delay a previous result until after a newer event
Documented transition rules preserve the correct outcome
Receipt failure after charge
Reject message delivery
Payment remains correctly recorded; notification retry is separate
UI success too early
Delay server completion
Displayed outcome follows authoritative payment state
Sensitive diagnostics
Trigger errors with synthetic protected values
Logs and error outputs exclude prohibited data
Critical limitation: Application tests alone cannot establish exactly-once charging across a
remote gateway. The claim depends on the gateway's idempotency or lookup behavior, local concurrency control, durable
identities, and reconciliation. Test the supported guarantee; do not invent a stronger one.
A useful mix is fast business-rule tests, database-backed transaction tests, controlled gateway-boundary tests, and
a small number of representative end-to-end sandbox checks. Counts and ratios should follow risk.
Repair-order or event synchronization
Requirement: Repeated and out-of-order events converge to the correct record under the actual
provider contract.
Case
What to test
Same event twice
No duplicate line, notification, or downstream effect
Older event arrives later
Explicit version/order rules; do not blindly assume timestamps totally order events
Partial payload
Defined meaning of omitted fields versus explicit null or deletion
Full snapshot replaces state
Correct removal and preservation rules
Provider unavailable
Bounded retries and visible lag
One malformed record in a batch
Documented isolation or failure policy; no silent loss
Worker restart mid-batch
Safe restart and accurate checkpoint handling
Mapping changes
Tenant and external identifiers remain correctly associated
Replay after code upgrade
Compatibility and deterministic transformation where promised
Apparent convergence
Reconcile field-level meaning and invariants, not just record counts
AI-written code and AI systems
Two different problems need different evidence.
When AI writes conventional code: Human-approved requirements and independent examples define
success. Tests derived only from generated implementation can reproduce the same misunderstanding. Feathers'
executable-constraint approach directly addresses that risk. S31
When AI is part of the running system (my recommended extension): Exact output strings
may be an inadequate oracle. Evaluate completed tasks, prohibited actions, uncertainty, latency, and cost over
representative cases. Use deterministic checks for tool authorization and data invariants; use repeated evaluation
where model behavior varies. Retain held-out cases and review grader errors.
Practical examples:
An email agent should be checked for the intended recipient, correct content, and whether sending was authorized, not only for fluent text.
A research agent should be checked against source evidence and citation support, not only an answer-shaped response.
A code agent should not receive a green result after silently deleting the failing acceptance test.
A tool-using agent should be tested with stale state, partial tool failure, cancellation, duplicate invocation, and untrusted instructions in retrieved content.
A local assistant should expose failures in speech recognition or downstream action rather than claiming the action succeeded.
These are proposed evaluation standards, not a claim that any finite evaluation makes an autonomous
system universally reliable.
7. What not to turn into dogma
Slogan
More defensible standard
"100% coverage means tested."
Ask what was asserted and which plausible faults survive.
"Every test must have one assertion."
Keep one coherent purpose; assert its important consequences.
"Never touch the database."
Isolate logic when useful; exercise actual persistence semantics when relevant.
"Mock every collaborator."
Control dependencies deliberately and validate real boundary assumptions.
"Never mock anything."
Use substitutes for fault control, speed, or otherwise inaccessible conditions.
"TDD proves correctness."
TDD is a development feedback workflow with prerequisites and limits.
"All failures are product bugs."
Diagnose product, harness, environment, and specification failures separately.
"A retry fixed the test."
A later pass does not explain the earlier failure.
"Snapshots are specifications."
A reviewed baseline may encode a decision; an automatically accepted one merely records output.
"End-to-end tests cover everything."
They sample workflows and often localize faults poorly.
"The simulator passed, so the device is safe."
Establish model fidelity and test physical assumptions where necessary.
"More tests always mean more confidence."
Value depends on independence, relevance, oracle strength, and maintenance cost.
8. A minimal record for a consequential test
Field
Record
Requirement or risk
What could go wrong and why it matters
Scope
Component, interface, workflow, deployment, or whole system
Preconditions
State, identities, configuration, environment
Stimulus
Inputs, sequence, concurrency, or injected failure
Oracle
Expected outcomes and the independent basis for them
Forbidden effects
What must not happen
Evidence
Actual observations, artifact identity, relevant logs
Section 17 of AGENTS.md distills this page into rules every coding agent here follows:
expected results that don't come from the code under test, the rules for AI-written code (standards 156 to 158),
honest reporting, and the release gate above. Our default tools stay Minitest and Rails' own test setup (AGENTS.md
section 5).
Research notes and source register
Research used author writing, first-person interviews, authorized publisher excerpts, research papers, and official
engineering guidance. Some author-site pages were unavailable, so the Meszaros analysis relies on the publisher's
sample chapters and contents. The Myers discussion is intentionally limited to accessible authorized previews rather
than presenting the entire book as reviewed. Mutable web references were consulted for this report; historical case
studies are not assessments of current product versions.
The named practitioners offer engineering judgment, not a single experimentally proven universal doctrine. The
standards combine their documented concerns with explicit synthesis. The sources below support the stated
foundations; the release gate and worked scenarios are original applications.
S1: Beck and Sutton, Test Desiderata
S2: Beck, TDD Prerequisites (2023)
S3: Feathers, Characterization Testing (2016)
S4: Feathers, Looking Back at Working Effectively with Legacy Code, first-person interview (2021)
S5: Meszaros, xUnit Test Patterns, publisher description and contents (2007)
S6: Meszaros, Refactoring a Test, publisher sample
S7: Meszaros, Test Double Patterns, chapter 23, publisher sample
S8: Myers, Sandler, Badgett, Thomas, The Art of Software Testing, 2nd ed., Software Testing Principles, authorized preview (2004)
S9: Same authors, The Psychology of Testing, authorized preview (2004)
S10: Fowler, Mocks Aren't Stubs (2007)
S11: NASA, Systems Engineering Handbook, Product Verification
S12: Perry and Luebbe, Google SRE, Testing for Reliability (2016)
S13: Myers, Sandler, Badgett, The Art of Software Testing, 3rd ed., Test-Case Design, chapter abstract
S14: Google SRE, Handling Overload
S15: Google SRE, Data Integrity: What You Read Is What You Wrote
S16: Jepsen, Consistency reference
S17: FoundationDB, Simulation and Testing, project documentation
S18: Claessen and Hughes, QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs (2000)
Provided as is, for information only, with no warranty and no liability. Not legal or professional advice. Security and radio material is for authorized, lawful use only. Full disclaimer.
Visits are logged on our own server: IP address, browser and device details, and pages viewed. We use them to understand traffic and spot attacks, and never share or sell them.