All articles

OpenAI strict structured outputs can't decline: check input in code

If you use OpenAI Structured Outputs with strict: true, the model cannot answer outside your schema, so a prompt rule like "if there is too little data, reply with {"unanswered": true}" can never be followed. On thin input the model fills every required property anyway, and you get a confident answer built on nothing. Enforce the minimum in code before the call, and delete the rule from the prompt so the two never disagree again.

The situation

A feature I work on turns a batch of short text snippets about a company into a list of positive and negative "drivers", the themes people mention most. The response comes back through Structured Outputs with a strict JSON schema: an object with arrays of drivers, every property listed in required, and additionalProperties: false, which is what strict mode demands.

The system prompt had grown up before the strict schema arrived. Near the end it carried a suppression rule: if the input has fewer than a certain number of snippets, do not produce the sections, and output a small JSON sentinel instead. Tests asserted that the rule was present in the prompt text. Everything looked deliberate and covered.

Why the sentinel can never come back

Strict mode constrains decoding to the schema. The sentinel object has one property, unanswered, which the schema does not declare, and it lacks the required arrays the schema does declare. There is no token sequence that satisfies both the prompt rule and the schema, and the schema wins every time, because it is enforced by the sampler and the prompt is only a request.

So on a thin input the model does the only thing it is allowed to do: it fills the arrays. With three snippets to work from, it produces a full, well-formed drivers list, and nothing in the response tells you it was stretched. That is worse than an error. A parse failure would have been noticed; a plausible invented answer gets stored and shown.

OpenAI's responses do have a refusal field, but that is for the model declining on safety grounds. It is not a channel for "I don't have enough data", and you should not design around it.

Where the check belongs

There are two honest ways out. One is to give the schema a legal way to decline, for example a boolean such as insufficient_evidence next to arrays that may be empty, and let the model decide. The other is to decide before the call.

We chose the second, for three reasons:

  • The rule was a count. Counting non-empty strings is something code does exactly and a model does approximately.
  • A call that can only produce an unusable answer is not worth paying for. Skipping it saves the tokens and the latency.
  • A check in code can be unit tested at the boundary: one below the minimum is rejected, the minimum itself is accepted. A prompt rule can only be tested by asserting that some text is present, which is exactly the test that had been passing all along while the behaviour was impossible.

The batch job that generates these lists already skipped the call below the minimum, with a comment saying the constant had to match the number in the prompt. That coupling was the smell. Once the prompt rule went, the constant stood on its own as the evidence floor, defined once in code.

The same floor also went into the HTTP endpoint that other services call for this generation, so a direct caller gets a 400 rather than an invented list. Here is the shape of it as a minimal ASP.NET Core endpoint:

const int MinimumEvidence = 5;

app.MapPost("/drivers", async (DriversRequest request, IDriversGenerator generator, CancellationToken ct) =>
{
    int usable = request.Snippets.Count(s => !string.IsNullOrWhiteSpace(s));
    if (usable < MinimumEvidence)
    {
        return Results.BadRequest(
            $"At least {MinimumEvidence} non-empty snippets are needed for a drivers list.");
    }

    DriversResponse drivers = await generator.GenerateAsync(request, ct);
    return Results.Ok(drivers);
});

public sealed record DriversRequest(IReadOnlyList<string> Snippets);

Two details matter in that check. It counts non-blank entries, not the array length, because an array padded with empty strings or whitespace is still thin evidence. And the tests feed it both cases: a list one short of the floor, and a list one short of the floor topped up with blank entries.

Scope the floor to the task that needs it

The same service exposed two other generations over the same snippets, each producing a single short insight rather than a ranked list. Those are reasonable on a handful of snippets, so the floor was deliberately not applied to them, and a test pins that split: thin input is a 400 for the drivers list and still a 200 for the single insights. Without that test, the next person to tidy the controller would very likely "fix the inconsistency" by applying the floor everywhere.

Clean the prompt, and keep it clean

Removing the rule was not quite enough. The output section of the prompt was headed "Output format rules (when answered)", which still implied a case in which the model declines. That qualifier went too.

The old tests that asserted the sentinel text was present were deleted, and one test replaced them: for every variant of the prompt, assert that it does not mention the sentinel, the suppression rule, the old threshold wording or "when answered". It is a small guard, but prompts get edited by people reading them as prose, and an opt-out rule that sounds sensible will come back unless something fails when it does.

Count failures as failures

A related fix landed in the same piece of work. When the generation call itself failed, the caller returned an empty string, which downstream code recorded the same way as "not enough evidence". The two outcomes looked identical in the data and in the metrics. A failed call now throws, so it reaches error tracking and the error count, and "skipped for thin evidence" means only that. The HTTP client for these calls also got an explicit timeout and retry policy instead of inheriting the defaults.

What I'd check in any structured-output prompt

Read the prompt and the schema side by side and look for any instruction that asks for a shape the schema cannot produce: a sentinel object, "reply with nothing", "say you don't know", a free-text apology. Each one is either dead text or, worse, a reader's false belief about how the system behaves. Then decide, case by case, whether the decline belongs in the schema as a declared field or in code before the call.

Key takeaways

  • With OpenAI Structured Outputs in strict mode, every response matches the schema, so a prompt rule asking for any other shape is silently impossible.
  • On thin input a strict schema produces a full, plausible answer rather than a refusal, which is harder to notice than an error.
  • Deterministic preconditions such as a minimum number of inputs belong in code before the call, where they are exact, free and testable at the boundary.
  • If the model genuinely needs a way to decline, declare it in the schema as a field, not in the prompt as a sentinel.
  • Add a test that the prompt does not contain the retired rule, and keep "the call failed" separate from "the call was skipped" in your data.