OpenAI JSON mode needs the word json: guard prompts from Langfuse
OpenAI's JSON mode (response_format: { "type": "json_object" }) rejects any request whose messages don't contain the word "json". When the prompt text lives in code, you notice once and never again. When it lives in Langfuse, anyone who edits the prompt can remove that word, and the next request fails at the API. The fix is a small guard: after loading and compiling the prompt, check it mentions JSON before calling the model, fail with the prompt's name and version when the word is missing, and keep that check in one shared helper so every JSON-mode call site gets it.
Why moving prompts out of code changes the risk
I've been moving a .NET service's LLM prompts out of C# constants and into Langfuse, where they're resolved at runtime by label. That's the point of a prompt manager: people can improve the wording without a deploy. It also means the prompt text is no longer reviewed alongside the code that depends on it.
Most of what a call needs from its prompt is loose. A reworded instruction produces a slightly different answer, and that's fine. A few things are hard contracts, though, and JSON mode's keyword rule is one of them. The API checks it before the model runs, so a prompt that lost the word doesn't produce a worse answer; it produces no answer at all, just a 400 from OpenAI that says the messages must mention JSON.
Someone tidying a system prompt can easily cut "Respond in JSON." because the response format already says so. It reads as redundant. It isn't.
Which calls need the guard
Only calls that use json_object mode. Structured Outputs with a strict JSON schema (response_format: { "type": "json_schema", ... }) has no keyword rule: the schema defines the shape, and the prompt can say whatever it likes. The same service made both kinds of call, some narrative calls with a strict schema and some insight calls in plain JSON mode, so the check had to be about the prompt text and applied only where JSON mode is used. Putting it inside the generic "call the model" path would have rejected perfectly valid strict-schema prompts.
That's also why I keep the predicate about the prompt alone. It answers "does this prompt mention JSON?", and each call site decides whether that matters for the response format it sends.
How the guard looks in .NET
Here's a minimal version with the official OpenAI .NET SDK. CompiledPrompt stands in for whatever your Langfuse loader returns once variables are substituted: the system and user text, plus the name and version you fetched.
using OpenAI.Chat;
public sealed record CompiledPrompt(string Name, int Version, string SystemPrompt, string UserPrompt);
public static class CompiledPromptExtensions
{
// OpenAI rejects json_object mode unless a message contains the word "json".
public static bool MentionsJson(this CompiledPrompt prompt) =>
prompt.SystemPrompt.Contains("json", StringComparison.OrdinalIgnoreCase)
|| prompt.UserPrompt.Contains("json", StringComparison.OrdinalIgnoreCase);
}
public sealed class InsightClient(ChatClient chat)
{
public async Task<string> GetInsightsAsync(CompiledPrompt prompt, CancellationToken ct)
{
if (!prompt.MentionsJson())
{
throw new InvalidOperationException(
$"Prompt '{prompt.Name}' v{prompt.Version} must mention JSON to run in JSON mode.");
}
ChatCompletionOptions options = new()
{
ResponseFormat = ChatResponseFormat.CreateJsonObjectFormat(),
};
ChatCompletion completion = await chat.CompleteChatAsync(
[new SystemChatMessage(prompt.SystemPrompt), new UserChatMessage(prompt.UserPrompt)],
options,
ct);
return completion.Content[0].Text;
}
}
In the real service the failure is a result value rather than an exception, but the shape is the same: check, then call.
Three details matter more than the code:
- Check after compiling. The word can legitimately arrive through a variable, so check the text you're about to send, not the template.
- Name the prompt and version in the error. "Prompt X v14 must mention JSON" points straight at the edit that broke it. A raw 400 from the provider, logged three layers down, doesn't.
- Fail before the call. You don't spend a request, and the error is yours, so it reads the same whichever provider or proxy sits in front of the model.
The substring match is deliberately crude. It guards against the word being removed, which is the realistic failure, not against a prompt that mentions JSON and still asks for prose.
Why the guard has to be shared
The first version of this guard was a private method on the one service where I'd hit the problem. Shortly after, migrating more prompts, I found another service that also sent JSON mode with a Langfuse-managed prompt and had no check at all. Nothing was broken yet, because nobody had edited that prompt. It just had the gap.
So the predicate moved into a shared extension on the compiled prompt type, and both call sites use it. That's the general lesson: a contract between an API and remotely edited text belongs next to the prompt type, not inside whichever service noticed it first. Otherwise every new JSON-mode call is one more place someone has to remember.
The same idea for schema fields
The JSON keyword isn't the only contract a remote prompt can break. If the prompt's response schema also lives in Langfuse and someone renames a property, System.Text.Json will still happily deserialise the model's answer; the field your code binds to just comes back empty. No exception, just blank output in the product.
The guard there is similar: before calling the model, check that the schema declares every property name the code reads, and fail with the prompt name and version if one is missing. It's a few lines of walking properties in a JsonElement, and it turns a silent empty value into a clear error.
How to test it
Two tests cover it. One feeds the service a prompt without the word and asserts a failure that mentions JSON. The other asserts the model was never called: with Moq, a MockBehavior.Strict mock of ChatClient does that for free, because any call it wasn't set up for throws. I added the same "model never called" assertion for the case where the prompt can't be loaded at all, since that's the other way a remote prompt store can fail you.
Key takeaways
- OpenAI's json_object mode rejects requests whose messages never mention JSON, so a prompt edited in Langfuse can break a call without any code change.
- Check the compiled prompt text for the word before calling the model, and only for json_object calls, because strict json_schema calls have no such rule.
- Put the check in one shared helper on the prompt type, since a guard that lives in one service will be missing from the next one that uses JSON mode.
- Name the prompt and its version in the error, so the failure points straight at the edit that caused it.
- Treat a remotely managed response schema the same way, and verify it still declares the properties your code reads before you send it.