Metamorphic Testing for AI Apps: Check Relationships Between Outputs
By Xusheng YangAn AI feature extracts invoice facts. The same invoice with different line breaks produces a different supplier identifier. Comparing complete answer strings would be too strict, but ignoring every difference would miss a real failure.
Metamorphic testing checks a relationship between executions on related inputs. You define a transformation that should preserve or change a particular property, then compare the relevant outputs under those conditions.
This is part 16 of Testing Methods in Practice. The invoice extractor is a fictional example with controlled fixtures, not a measured claim about an AI model or HiTest.
Choose a property with a defensible relationship
Suppose the feature extracts supplier ID, currency, total, and invoice date into structured fields. Prepare a synthetic invoice with known values and no personal data. Define how ambiguity, missing values, and invalid invoices should be represented.
For the initial relation, change only harmless formatting: add blank lines or adjust spacing between sections without changing the text or reading order. The extracted supplier ID and other agreed facts should remain equal after normalization.
The research review Metamorphic Testing: A Review of Challenges and Opportunities describes testing through relations when direct result verification is difficult. The hard part is choosing a relation that is valid for the feature, not just easy to automate.
Write transformations with preconditions
| Transformation | Relationship to check | Important precondition |
|---|---|---|
| Add blank lines | Extracted facts remain equal | Layout change does not alter reading order or labels |
| Change irrelevant decorative text | Protected invoice facts remain equal | Text is genuinely outside the extraction contract |
| Replace the total with another valid total | Extracted total changes to the new known value | All dependent amounts are updated consistently |
| Remove the supplier identifier | Output follows the documented missing-value policy | No other source contains that identifier |
| Add a second ambiguous invoice | Output follows the multi-document or ambiguity policy | The product explicitly defines that input class |
Do not assume arbitrary sentence shuffling is harmless. Order can change meaning. Do not assume translating a legal or financial term preserves the feature's interpretation unless that is part of its contract.
Keep ordinary expected-value checks
For the small synthetic invoice, you already know the correct fields. Check those values directly as well as the relationships. Two outputs that are identically wrong satisfy an equality relation.
Metamorphic cases add coverage for transformations. They do not replace known-answer fixtures, authorization checks, or an independent review of extraction quality.
Use a negative control: intentionally change the supplier ID and verify that the extractor responds to the changed fact. If it always returns the original fixture's value, a “formatting invariance” check alone would incorrectly look reassuring.
Handle nondeterminism explicitly
Record the model or provider version when available, prompt version, extraction schema, parameters, transformation, and fixture identity. Repeat each original/transformed pair enough times to inspect variation under the same configuration.
Choose the comparison before collecting results. Exact equality may be appropriate for a normalized identifier. A narrative explanation may require a rubric, source-grounding checks, or human judgment instead of string equality.
Report sample counts and observed violations. Do not choose a pass threshold after seeing the failures or turn a small sample into a claim about all future runs. If an AI judge is used, treat its verdict as another fallible observation and review disagreements against the source document.
Test the application around the model
The service may extract correctly while the UI displays fields from a previous upload. Upload original and transformed files with distinguishable names, verify which document each result belongs to, reload, and inspect retained results.
Check cancellation, retry, and missing-field presentation using deterministic stubs where appropriate. Keep live-provider evaluations separate from repeatable UI state tests so a provider variation does not obscure a browser regression.
For cost-bearing evaluations, establish a deliberate run budget and use synthetic or approved test data. The test method itself does not require sending sensitive invoices to a provider.
A reusable relation worksheet
Record the source input, transformation, preconditions, expected relation, direct facts you can verify, comparison method, repeated-sample plan, and negative control. Mark which layer observes each result.
Your exercise: propose three relationships for an AI feature, then try to disprove each with a counterexample. Keep only relationships justified by the feature's requirements. Use assertion design to select the evidence and equivalence partitioning to organize distinct input classes.
Next in the series: Time-dependent testing.