After switching models, check response completion first, schema conformance second and business facts third. Successful JSON parsing does not establish that an amount, currency or reference number came from the input, nor does it authorize a downstream write.
Official documentation was reviewed on 2026-10-08. The quotes and outputs below are synthetic fixtures, not paid model runs or accuracy rankings. This guide evaluates returned data; provider pinning and bounded retries cover different parts of the integration.
Identify the actual endpoint contract
Record the model, processing provider, API path, SDK version, request settings and schema version. Changing only the model name does not prove that the rest of the behavior stays constant.
| Approach | Purpose | Application checks still needed |
|---|---|---|
| Prompt requesting JSON | Describe the desired format | Extra prose, missing fields and wrong types |
| JSON mode | Request parseable JSON | Business schema and response completion |
| JSON Schema structured output | Apply structural constraints where supported | Endpoint-specific keyword support, exceptions and facts |
| Tool-call arguments | Propose parameters for an action | Tool identity, permission and business validation before execution |
OpenRouter’s reference describes support per endpoint and provider-dependent strict enforcement. Its structured-output request uses response_format.type: "json_schema"; provider.require_parameters: true can restrict routing to endpoints supporting required parameters. Routing eligibility does not replace application validation.
Inspect a redacted copy of the actual outgoing request. Check SDK schema transformations, gateway parameter removal and fallback routes. Unsupported configurations should fail explicitly or enter a separately validated fallback, rather than silently reverting to ordinary text and continuing automated writes.
Freeze a contract that distinguishes unknown from zero
Start with a narrow quote-extraction task: reference, currency, subtotal, tax and total. This example supports only CNY and USD, with nonnegative integer amounts in cents. Refunds, negative amounts and other currencies are outside its contract.
Input: Quote Q-17. Currency CNY. Subtotal 100.00; tax 6.00; total 106.00.
Expected output:
{
"quote_id": "Q-17",
"currency": "CNY",
"subtotal_minor": 10000,
"tax_minor": 600,
"total_minor": 10600,
"status": "complete"
}Require all six keys. Unstated values must be null and routed to needs_review, not replaced by zero, an empty string or a derived estimate. The model’s proposed status is not the application’s acceptance decision.
The application schema below is available as a Draft 7 schema file. It is not a universal provider request body. Where an API supports only a subset, document its adaptation and retain the full application validator.
{
"type": "object",
"properties": {
"quote_id": { "type": ["string", "null"] },
"currency": { "enum": ["CNY", "USD", null] },
"subtotal_minor": { "type": ["integer", "null"], "minimum": 0 },
"tax_minor": { "type": ["integer", "null"], "minimum": 0 },
"total_minor": { "type": ["integer", "null"], "minimum": 0 },
"status": { "enum": ["complete", "needs_review"] }
},
"required": ["quote_id", "currency", "subtotal_minor", "tax_minor", "total_minor", "status"],
"additionalProperties": false
}Requiring a key and permitting a null value are compatible choices. The JSON Schema object reference distinguishes missing properties from null values; listing a field in properties alone does not make it required.
Apply the same negative fixtures to both integrations
Download the 10 synthetic acceptance fixtures. First exercise your validator with their outputs, then use their source texts as inputs to the old and new integrations. The supplied output objects are hand-authored test data, not observed model responses.
| Change from the valid example | Schema result | Application disposition |
|---|---|---|
| All fields match the source | Pass | Accept only after business checks |
Remove tax_minor |
Fail | Report missing key; do not insert zero |
Null tax with status complete |
Pass | Reject the completion claim |
String total "10600" |
Fail | Do not silently coerce types |
Add approved: true |
Fail | Reject an unknown authorization field |
| Currency EUR | Fail | Outside this example’s supported set |
| Total 10700 | Pass | Reject arithmetic and source mismatch |
| Currency USD | Pass | Allowed value, but wrong for this source |
| Negative tax | Fail | Outside the nonnegative quote contract |
Source omits tax; null tax and needs_review |
Pass | Preserve uncertainty; no automatic publishing |
Business checks also require a nonempty reference matching the source, correct units and subtotal_minor + tax_minor = total_minor. Evaluate the equation only when all three values are known. Being able to infer a tax amount does not make it an extracted source fact.
Count structurally valid but factually wrong outputs separately. Use human-labeled source facts for acceptance; production facts that cannot be checked reliably need a review path. A second model’s approval does not substitute for deterministic arithmetic.
Handle refusal and truncation before field validation
An adapter should inspect API errors, refusals, finish reasons, empty content and stream completion before extracting the final data block for parsing.
For example, Claude’s structured-output documentation states that stop_reason: "refusal" can accompany HTTP 200 without schema-conforming output, and max_tokens can leave output incomplete. Those field names belong to that API. Map each service’s response contract instead of assuming every provider uses them.
| Response condition | Application branch | Avoid |
|---|---|---|
| Unsupported parameters or API error | Configuration failure | Removing constraints and calling it a pass |
| Explicit refusal | Separate refusal state | Fabricating an object to bypass it |
| Output limit or interrupted stream | Incomplete result | Adding closing braces and writing partial data |
| Empty content or parse failure | Parsing failure | Returning another task’s cached result |
| Schema mismatch | Record field path and violated rule | Silent field deletion or default insertion |
| Business mismatch | Review or failure | Trusting the model’s complete flag |
Inject refusal, truncation and empty-content envelopes into adapter tests. These validate application branches, not real model refusal rates. Any bounded repair must retain original and repaired results, pass every check again and consume the shared retry budget.
Compare migrations by failure category
Freeze source texts, schema, prompt and acceptance rules, then change one variable at a time. If a model requires different settings, record that as the actual migration configuration rather than claiming identical conditions. Include ordinary, incomplete, contradictory, long and unrelated inputs; do not select only favorable examples.
Keep a row per attempt: sample ID / model and endpoint / SDK and config version / schema version / request ID / completion state / parse result / schema result / factual result / final disposition / elapsed time / confirmed usage. Report recovery after retries separately from first-attempt acceptance.
| Measure | Denominator or interpretation |
|---|---|
| First-attempt business acceptance | All submitted samples; do not silently exclude refusals or truncation |
| Parsing and schema failures | Also report responses stopped before these stages |
| Schema pass with business failure | Exposes false confidence from valid structure |
| Review, refusal and incomplete counts | Separate categories with business-specific tolerance |
| Recovery and unresolved cost counts | Preserve recovery costs; unknown cost is not zero |
These ten fixtures are a regression starting point, not a statistical model-accuracy benchmark. Set blocking criteria before rollout: for example, any incorrect amount entering automated publishing blocks this migration. Lower-risk extraction may retain human review. Start with limited use, preserve the old configuration and rollback path, and expand only with evidence from actual failures.
For a general small-set comparison when changing a prompt, see the ten-case prompt evaluation method. It does not replace the schema and factual checks on this page.