Stop OpenSearch .NET client failures caching as empty results

The OpenSearch .NET client (OpenSearch.Client 1.x) does not throw when a search fails with a 5xx or a timeout by default. It returns a response with IsValid set to false and empty hits and aggregations. If your code reads those with null-conditional operators and returns an empty list, and a cache sits above it, one failed search becomes a cached "nothing matches" for the whole cache lifetime. The fix is to check IsValid right after every search, throw, and let the exception travel past every cache and every catch-all until it becomes an error response.

The feature that exposed it

I was adding a "filter by job title" option to a company directory on a job board platform. Two .NET services are involved. One owns the jobs index and answers "which companies have at least one job on this site matching this title?" with a terms aggregation on the company id. The other owns the companies index; it takes those ids and adds them to the non-scoring filter clause of its own bool query, so totals and paging reflect the filtered set rather than filtering a page after the fact.

A second endpoint returns up to three matching job titles per company for the directory cards. Both endpoints sit behind a Redis cache, because the same popular titles are asked for over and over.

The first version worked in every test. Code review found the problem: every failure path in the chain quietly turned into a valid, empty, cacheable answer.

Why a failed search looked like an empty one

There were three layers, and each one was reasonable on its own.

1. The search code read response.Aggregations?.Terms("companies")?.Buckets and returned an empty list when that was null. A failed response has no aggregations, so a 503 from the cluster came back as "no companies".
2. The caching decorator stored whatever the inner service returned. An empty list is a perfectly good value to cache.
3. The calling service wrapped its search in a catch-all that logged and returned an empty page, which is the right instinct for an optional widget. Here it was the main query of the page, so a timeout produced an HTTP 200 with zero results, which any HTTP cache in front would happily keep.

The user sees "no companies are hiring for this title" until the cache entry expires, and nothing in the logs says the answer was wrong, because nothing failed from the code's point of view.

How to make the client fail loudly

There is a global switch, ConnectionSettings.ThrowExceptions(), that makes the low-level client throw on any failed call. It is tempting, but in a service that already has code relying on the non-throwing behaviour, flipping it changes every existing call at once. I preferred a small guard used by the new searches:

using OpenSearch.Client;

static void EnsureValid<T>(ISearchResponse<T> response, CancellationToken ct)
    where T : class
{
    if (response.IsValid)
        return;

    // An aborted request also surfaces as an invalid response; report it as a cancellation.
    ct.ThrowIfCancellationRequested();

    throw new InvalidOperationException(
        $"OpenSearch search failed with status {response.ApiCall?.HttpStatusCode}.",
        response.OriginalException);
}

public async Task<IReadOnlyList<long>> CompanyIdsForTitleAsync(
    IOpenSearchClient client, string title, CancellationToken ct)
{
    var response = await client.SearchAsync<JobDocument>(s => s
        .Size(0)
        .Query(q => q.Match(m => m.Field(f => f.Title).Query(title).Operator(Operator.And)))
        .Aggregations(a => a.Terms("companies", t => t.Field(f => f.CompanyId).Size(10_000))),
        ct);

    EnsureValid(response, ct);

    return response.Aggregations.Terms("companies").Buckets
        .Select(b => long.Parse(b.Key))
        .ToList();
}

Two details matter. The cancellation check comes before the throw, so a client that disconnected is reported as a cancellation, not as a cluster failure that pages someone. And any surrounding try/catch that logs and rethrows needs a catch (OperationCanceledException) { throw; } ahead of the general handler, for the same reason. I pinned both with tests: an invalid response throws, and an invalid response with a cancelled token throws OperationCanceledException.

Where the lookup should live in the caller

Throwing in the search service is only half of it. In the calling service the job-title lookup had been made inside the repository, inside the catch-all, and was called once for each of the two queries that build the page. I moved it up a level: the service resolves the hiring company ids once per request, outside the catch-all, and passes the same ids to both queries. A failure in the other service now propagates to a 500, which nothing caches, instead of an empty 200, which everything does.

That also removed a dependency from the repository, which had been talking to another service over HTTP. The repository went back to building queries, which is all it should do.

One more cache detail: the job title is free text, and it became part of the Redis key. I normalised it (trim, lower-case), capped it at 100 characters with a 400 above that, and escaped it in the key so a title containing the key separator cannot collide with another key.

Three titles per company with terms and top_hits

The card endpoint is a nice use of a sub-aggregation. A terms aggregation on the company id, sized to the number of companies asked for, with a top_hits sub-aggregation that returns only the title field:

• When a title filter is applied, top_hits keeps its default score order, so the best matches come first.
• Without a filter, it sorts by creation date descending, so the newest jobs show.
• It asks for four times as many hits as it needs, because one company often posts the same title several times, and the code takes the first three distinct titles case-insensitively.

The id lookup uses a fixed bucket cap on its terms aggregation. That is a deliberate shortcut: past the cap, companies would be silently dropped, and the honest fix at that point is a composite aggregation with paging, not a bigger number.

A serializer surprise in top_hits documents

The top_hits documents came back with null titles at first. The service used a client configured with a System.Text.Json source serializer, and the document class mapped its fields with [DataMember(Name = "...")]. System.Text.Json ignores DataMember names, so _source fields with snake_case names never bound. Aggregation keys and counts are parsed by the client itself, which is why the id lookup worked and only the document reads failed. Reading those documents through a client configured with OpenSearch.Client.JsonNetSerializer, which honours the attribute, fixed it. If one service has two clients with different source serializers, it is worth a comment at the call site saying which one a read needs and why.

Key takeaways

• The OpenSearch .NET client reports 5xx responses and timeouts through IsValid rather than exceptions, so check it after every search you write.
• An empty result produced by a failure is worse than an error when a cache sits above it, because the wrong answer outlives the outage.
• Check the cancellation token before throwing on an invalid response, so aborted requests are not reported as cluster failures.
• Make cross-service lookups once per request and outside any catch-all that turns errors into empty pages.
• A System.Text.Json source serializer ignores [DataMember] names, so documents mapped that way need a serializer that honours them.