All articles

Moving a .NET OpenAI SDK call to a LiteLLM proxy: three silent changes

, 5 min read

When you move a .NET service from the official OpenAI SDK's ChatClient to a shared client that calls a LiteLLM proxy, three things can change without a compile error or a failing unit test: the request timeout, the reasoning effort the model runs at, and whether the proxy key is even allowed to call the model you asked for. Give the migrated call its own named HttpClient with its own timeout and its own virtual key, and send reasoning_effort on every request instead of relying on a default.

What the migration looked like

The service in question generates a structured summary with a reasoning model. It had been written early, before the team put a LiteLLM proxy in front of every model call, so it built its own ChatClient from an OpenAI API key and passed ChatMessage and ChatCompletionOptions objects around. Every other flow in the service already went through a small typed client that posts to the proxy's OpenAI-compatible /chat/completions endpoint and returns a parsed, schema-checked result.

The migration itself was mostly deletion. The prompt class stopped building SDK message objects and exposed its system text, tool name and JSON schema as plain values. The service took the proxy client from dependency injection, and the unit tests swapped a mocked ChatClient (with hand-built ChatCompletion JSON) for a mocked interface that returns the parsed result. The tests passed. The call still worked against the proxy. And three behaviours had quietly moved underneath it.

Why the timeout dropped from minutes to 100 seconds

The old client had been given a ceiling of three minutes, because a reasoning model producing a long structured answer can legitimately take that long. The shared proxy client was registered through IHttpClientFactory without setting a timeout at all, which means it inherited the HttpClient.Timeout default of 100 seconds.

Nothing about this shows up in a unit test that mocks the client interface. We caught it while reading the registration, before it shipped; left alone, it would have surfaced as a TaskCanceledException on the slowest requests in production, which are exactly the ones you can least afford to lose. Raising the timeout on the shared client is the wrong fix: the other flows are short and benefit from failing fast. The fix is a separate named client for the slow flow, registered the same way as the shared one but with its own timeout.

public static IHttpClientBuilder AddLlmProxyClient(
    this IServiceCollection services,
    string name,
    Func<LlmProxyOptions, string?> pickKey,
    TimeSpan? timeout = null)
{
    return services.AddHttpClient(name, (sp, http) =>
    {
        var options = sp.GetRequiredService<IOptions<LlmProxyOptions>>().Value;
        http.BaseAddress = new Uri(options.BaseUrl);
        http.DefaultRequestHeaders.Authorization =
            new AuthenticationHeaderValue("Bearer", pickKey(options) ?? options.DefaultKey);
        if (timeout is { } t)
        {
            http.Timeout = t;
        }
    });
}

// Short flows share the default; the summary flow gets its own ceiling and key.
services.AddLlmProxyClient("default", o => o.DefaultKey);
services.AddLlmProxyClient("summary", o => o.SummaryKey, TimeSpan.FromMinutes(3));

Routing the slow flow through the same helper, with an optional timeout, matters more than it looks. The first version of the fix registered the new client by hand, and review pointed out that it now had a second copy of the base address and auth wiring that could drift from the shared one.

Why reasoning effort fell from medium to low

The old code never set a reasoning effort. With the OpenAI API, leaving reasoning_effort out of a request to a reasoning model means the API applies its own default, which is medium. The proxy client, on the other hand, always sends a value: it reads it from a settings object, and the settings class defaults it to low, because the flows that were built for the proxy are quick classification calls.

So the migrated call went from "omitted, so medium" to "explicitly low". That is not an error. The model still answers, the JSON still validates, and the output is plausibly fine. It is just a different model behaviour than the one the prompt was tuned against, and nobody chose it.

The general lesson is that "the default" is not a single value when two layers each have one. If a call depends on a setting, send it explicitly on that request:

var request = new StructuredCompletionRequest
{
    Model = "gpt-5",
    SystemPrompt = SummaryPrompt.System,
    UserPrompt = prompt.User,
    ToolName = SummaryPrompt.ToolName,
    Schema = SummaryPrompt.Schema,
    ReasoningEffort = "medium" // the API's own default, now stated rather than inherited
};

A unit test that captures the request and asserts the effort, model, tool name and schema it sends is cheap, and it is the only kind of test that would have caught this.

Why the proxy refused the model

LiteLLM virtual keys can be restricted to a list of models. That is a sensible setup: each flow gets a key scoped to the aliases it uses, and spend and errors are attributed per key. The shared key in this service was locked to the alias the classification flows use. The migrated flow asked for gpt-5 with that key, and the proxy rejected the request as not allowed for this key.

This one does fail loudly, but only against a real proxy. A wiring test that resolves the named client from the DI container can at least check that the right key ends up in the Authorization header. The pattern that worked was a per-flow key setting that falls back to the shared key when it is not set, the same as the other flows already did. The fallback keeps local development simple, and production sets the scoped key.

The new key also had to reach the running service. In this case it lives in a secret store that infrastructure code reads at deploy time, and the lookup fails the deployment if the field does not exist yet. That ordering is worth keeping: a missing secret should stop the apply, not deploy a service that will be refused on its first call.

How to test the parts a mock hides

All three changes lived in configuration and wiring, below the interface the unit tests mock. Two cheap integration-style tests cover them in ASP.NET Core with WebApplicationFactory:

[Fact]
public void SummaryClient_HasItsOwnTimeoutAndKey()
{
    using var factory = new WebApplicationFactory<Program>()
        .WithWebHostBuilder(b =>
        {
            b.UseSetting("LlmProxy:DefaultKey", "shared-key");
            b.UseSetting("LlmProxy:SummaryKey", "summary-key");
        });

    var clients = factory.Services.GetRequiredService<IHttpClientFactory>();
    using var http = clients.CreateClient("summary");

    Assert.Equal(TimeSpan.FromMinutes(3), http.Timeout);
    Assert.Equal("summary-key", http.DefaultRequestHeaders.Authorization?.Parameter);
}

The second test is the request-capture one above: mock the client interface, let the service call it, and assert on the request object it built. Together they pin the timeout, the key and the effort, which were the three things that changed.

Key takeaways

  • A named HttpClient registered without a timeout uses the 100-second HttpClient.Timeout default, which is too short for many reasoning-model calls.
  • Give a slow LLM flow its own named client through the same registration helper as the others, rather than raising the shared timeout or duplicating the wiring.
  • When the OpenAI API and your own client each have a default for reasoning_effort, send the value you want on every request.
  • LiteLLM virtual keys scoped to specific models will refuse other models, so give each flow a key setting that falls back to the shared one.
  • Unit tests that mock the LLM client cannot see timeouts, keys or request defaults; add a DI wiring test and a request-capture test.