All articles

What to keep in code when moving LLM prompts to Langfuse

, 5 min read

When you move LLM prompts from compiled-in strings to Langfuse, move the instructions and leave in code anything that is built per request, which usually means the JSON schema for structured output. Also give the schema its own format name, because OpenAI's json_schema.name only accepts letters, digits, underscores and hyphens, so a Langfuse prompt name that lives in a folder (area/task) cannot be reused as that name.

Which parts of a prompt actually belong in Langfuse?

The appeal of a prompt-management service is that a prompt engineer can change wording, compare versions and roll back without a deployment. That only helps for the parts that are really static: the system instructions, the tone, the worked examples and the fixed scaffolding of the user message.

Many production prompts also contain parts that are not static. A classifier may need an enum of allowed values taken from a database that differs per tenant. A matching prompt may need the list of candidates the caller supplied for this one request. Those values appear in two places: in the text of the prompt, and in the structured-output schema that constrains the reply. A schema stored next to the prompt has to be a fixed document, and an enum built from the current request cannot be expressed as one.

So the split I settled on is simple. The text that a human would want to edit lives in Langfuse. Anything computed from the request stays in code and is appended or substituted at call time.

How to load a prompt and still build the schema per request

I wrap prompt loading behind a small interface that returns chat messages with variables already substituted. The shape is what matters:

public interface IPromptStore
{
    // Returns the chat messages for a prompt, with template variables substituted.
    Task<IReadOnlyList<ChatMessage>> GetAsync(
        string name, IReadOnlyDictionary<string, string> variables, CancellationToken ct);
}

public sealed record ChatMessage(string Role, string Content);

public async Task<string> ClassifyAsync(
    string itemDescription, IReadOnlyList<string> allowedLabels, CancellationToken ct)
{
    var messages = await _prompts.GetAsync(
        "classify/item",
        new Dictionary<string, string> { ["item"] = itemDescription },
        ct);

    // The enum depends on this request, so the schema is built here, not stored.
    var schema = new JsonObject
    {
        ["type"] = "object",
        ["additionalProperties"] = false,
        ["required"] = new JsonArray("label"),
        ["properties"] = new JsonObject
        {
            ["label"] = new JsonObject
            {
                ["type"] = "string",
                ["enum"] = new JsonArray(allowedLabels.Select(l => (JsonNode)l).ToArray()),
            },
        },
    };

    return await _llm.CompleteAsync(messages, ResponseFormat.JsonSchema("classify_item", schema), ct);
}

I also made the loader refuse a prompt that has no stored response schema unless the caller says that is expected. That default catches a prompt someone created in the UI and forgot to configure. The few prompts that build their own schema opt out explicitly, so the exception is visible in code review and in the documentation rather than being an accident.

Why can't the Langfuse prompt name be the schema name?

Langfuse lets you organise prompts in folders by putting a slash in the name. That is a good habit once you have more than a handful. But OpenAI's structured outputs take a json_schema object with a name field, and that name is restricted to a-z, A-Z, 0-9, underscores and dashes, with a length limit. A name like classify/item is rejected.

The temptation is to derive the format name from the prompt name by replacing the slash. Don't. It couples two things that have different rules and different owners, and a rename in the prompt UI would change what is sent to the API. Give the schema a constant of its own, as "classify_item" is above, and add a one-line test that it contains no slash. That test costs nothing and fails in the right place if someone later "tidies" the constants.

Do you need a compiled fallback?

A remote prompt store adds a runtime dependency. When the service is unreachable and nothing is cached, the call cannot be made. The usual answer is to keep a compiled-in copy of every prompt as a fallback, but that has a cost: two copies of each prompt drift apart, and the fallback is exactly the code path that is never exercised until the day it matters.

What I did instead was to look at who calls each prompt and what they do with a failure:

  • If the caller is a batch job that already records a failed item and moves on, a load failure is a failed item. Retrying later is correct, and a stale compiled prompt would only produce a classification nobody can trace to a prompt version.
  • If the caller is a background backfill that logs and carries on, the same applies.
  • If a user is waiting on the response, a fallback may be worth its cost.

For the prompts I was moving, none of the callers fell into the third case, so I removed the compiled text. The behaviour on failure is then a deliberate, tested contract: when the prompt cannot be loaded, the service returns an internal-server-error result and the model is never called. I added a test for exactly that, with a prompt store stub that throws, asserting both the result type and that the LLM client received no call. Without it, a refactor could quietly turn a load failure into a call with an empty prompt.

The in-process cache of last-known-good prompts covers an outage that begins after a service is warm. It does not cover a service that starts during one. That is worth a separate decision, such as warming every registered prompt at startup and logging how many resolved.

How to test the split

Two small habits kept the test suite honest. First, a shared stub prompt store, so every service test builds its chat messages the same way instead of each class hand-rolling a one-system-one-user fixture. Second, a test per prompt that asserts the request sent to the model has a strict schema and contains the compiled variables. Those pin the part of the contract that now lives across two systems.

Key takeaways

  • Put the text people want to edit in Langfuse, and keep in code anything computed from the request, such as an enum built from per-tenant data.
  • A stored response schema must be a fixed document, so prompts whose schema depends on the request should opt out of the schema requirement explicitly.
  • OpenAI's json_schema.name cannot contain a slash, so give each schema its own constant name rather than deriving it from a foldered Langfuse prompt name.
  • Decide on a compiled fallback by looking at what each caller does on failure; where a caller already handles failure, a tested error result is safer than a second copy of the prompt.
  • Test the failure path, asserting that a prompt that cannot be loaded returns an error and the model is never called.