All articles

Logging OpenTelemetry .NET exporter failures with an EventListener

, 5 min read

The OpenTelemetry .NET OTLP exporter does not throw or log when an export is rejected. It writes the failure to its internal EventSource and tells its caller everything went fine, so a broken trace pipeline can run for weeks with nothing in your logs. The fix is a small EventListener that forwards warnings and errors from the OpenTelemetry event sources into ILogger, plus a one-off probe at startup that shows the raw HTTP status.

Why does a failed OTLP export produce no log line?

I was sending LLM generations from a .NET service to Langfuse over OTLP. Traces appeared in testing, then, after an infrastructure change, they stopped. The service was healthy, requests were succeeding and the application log was clean. Nothing said that the exporter was being refused.

The reason is how the SDK is built. Tracing must never take down the application it observes, so the exporter swallows its own failures. It reports them through event sources named OpenTelemetry-Sdk, OpenTelemetry-Exporter-OpenTelemetryProtocol and similar. If nothing is listening to those sources, the message goes nowhere. That is a sensible default for a library and a poor one for a team that depends on the data.

How do I forward OpenTelemetry EventSource events to ILogger?

Subclass EventListener, enable any source whose name starts with OpenTelemetry at warning level, and write what arrives to your logger. Registering it as a hosted service keeps its lifetime tied to the host.

using System.Diagnostics.Tracing;
using System.Globalization;

internal sealed class OtelDiagnostics : EventListener, IHostedService
{
    private readonly ILogger<OtelDiagnostics> logger;

    public OtelDiagnostics(ILogger<OtelDiagnostics> logger) => this.logger = logger;

    public Task StartAsync(CancellationToken ct) => Task.CompletedTask;
    public Task StopAsync(CancellationToken ct) => Task.CompletedTask;

    protected override void OnEventSourceCreated(EventSource source)
    {
        if (source.Name.StartsWith("OpenTelemetry", StringComparison.Ordinal))
        {
            EnableEvents(source, EventLevel.Warning, EventKeywords.All);
        }
        base.OnEventSourceCreated(source);
    }

    protected override void OnEventWritten(EventWrittenEventArgs e)
    {
        if (logger is null) return;   // see the next section

        var level = e.Level switch
        {
            EventLevel.Critical or EventLevel.Error => LogLevel.Error,
            EventLevel.Warning => LogLevel.Warning,
            _ => LogLevel.Information,
        };

        var payload = e.Payload?.Select(p => p?.ToString() ?? "").ToArray() ?? [];
        var detail = string.IsNullOrEmpty(e.Message)
            ? string.Join(" | ", payload)
            : string.Format(CultureInfo.InvariantCulture, e.Message, payload);

        logger.Log(level, "OpenTelemetry {Source}/{Name}: {Detail}",
            e.EventSource.Name, e.EventName, detail);
    }
}

Register it after AddOpenTelemetry with builder.Services.AddHostedService<OtelDiagnostics>(). Hosted services start in registration order, and you want the tracer provider already running when the listener comes up.

What goes wrong with an EventListener constructor?

The base EventListener constructor calls OnEventSourceCreated for every event source that already exists, and it does so before your own constructor body has run. Any field you assign in your constructor is still null at that moment. That is why the example above touches no instance state inside OnEventSourceCreated, and why OnEventWritten guards against a null logger: an event can arrive on another thread during construction.

Format the message defensively too. string.Format with an event's message template and its payload can throw a FormatException if they disagree, and an exception escaping a diagnostics listener is the last thing you want. Catch it and fall back to the joined payload.

How do I see the real status code behind an export?

The listener tells you the exporter failed, but sometimes the message is vague. In my case the interesting failure was an authentication proxy sitting in front of the endpoint. The exporter's own HttpClient followed its redirect to a login page, received a 200 and treated the export as a success. Nothing was reported as wrong because, from the client's point of view, nothing was.

A startup probe settles this. Send one request to the OTLP endpoint with automatic redirects disabled and log the raw status and Location header:

var handler = new HttpClientHandler { AllowAutoRedirect = false };
using var http = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(10) };

using var request = new HttpRequestMessage(HttpMethod.Post, tracesUrl)
{
    Content = new ByteArrayContent([]),
};
request.Content.Headers.ContentType = new("application/x-protobuf");
// add the same auth headers the exporter uses

using var response = await http.SendAsync(request, ct);
var level = (int)response.StatusCode switch
{
    >= 200 and < 300 => LogLevel.Information,
    >= 300 and < 400 or 401 or 403 => LogLevel.Error,
    _ => LogLevel.Warning,
};
logger.Log(level, "ingest check POST returned {Status} Location={Location}",
    (int)response.StatusCode, response.Headers.Location);

I treat redirects, 401 and 403 as errors, because they mean the credentials or the route are wrong. A 400 on an empty body is only a warning: the endpoint was reached and the probe payload was simply not valid. A timeout or connection failure is logged and swallowed, since a probe must never stop the host from starting. If the credentials are still a deployment placeholder, skip the probe and say so in the log.

The probe is also the place to emit one generation through the real instrumentation path. That proves the activity source, the tracer provider and the exporter are wired together, which a direct HTTP call alone cannot.

How do I test an EventListener?

You do not need OpenTelemetry in the test. Declare a private EventSource whose name starts with OpenTelemetry, write an error event from it, and assert that a capturing ILogger received it at error level. Add a second source with an unrelated name and assert that its events are ignored. Both tests run in milliseconds and pin the prefix match, which is the part most likely to be changed by accident.

Key takeaways

  • The OpenTelemetry .NET exporters report their own failures only to EventSource, so with no listener a rejected export leaves no trace in your logs.
  • An EventListener that enables sources starting with OpenTelemetry at warning level and writes to ILogger is about sixty lines and makes those failures visible.
  • Do not use instance fields from OnEventSourceCreated, because the base constructor calls it before your constructor body has assigned them.
  • A redirect from an authentication proxy can turn into a successful-looking export, so probe the endpoint once with redirects disabled and log the raw status.
  • Register the listener and the probe after AddOpenTelemetry, since hosted services start in registration order.