OpenTelemetry

Analyze your software performance by instrumenting, generating, collecting, and exporting telemetry data.

Module Overview

The module provides OpenTelemetry distributed tracing support. The module supports W3C context propagation and OTLP/gRPC export protocol.

Note:
the code of NGINX OpenTelemetry module is open source since NGINX Open Source 1.25.2 and NGINX Plus Release 30. The source code is available on GitHub.

Installation

  1. Check the Technical Specifications page to verify that the module is supported by your operating system.

  2. Install the OpenTelemetry module package nginx-plus-module-otel.

    For CentOS, Oracle Linux, and RHEL:

    yum install nginx-plus-module-otel

    For Amazon Linux 2023, AlmaLinux, Rocky Linux:

    dnf install nginx-plus-module-otel

    For Debian and Ubuntu:

    apt-get install nginx-plus-module-otel

    For SLES:

    zypper install nginx-plus-module-otel

    For Alpine:

    apk add nginx-plus-module-otel

    For FreeBSD:

    pkg install nginx-plus-module-otel
    Note:
    the OpenTelemetry module cannot be installed on RHEL/Oracle Linux/AlmaLinux/Rocky Linux 7, Ubuntu 18.04, and Amazon Linux 2.

Configuration

After installation you will need to enable and configure the module in NGINX Plus configuration file nginx.conf.

  1. Enable dynamic loading of the module with the load_module directive specified in the top-level (“main”) context:

    load_module modules/ngx_otel_module.so;

  2. Test the configuration and reload NGINX Plus to enable the module:

    nginx -t && nginx -s reload

Module directives

otel_exporter

Syntax: otel_exporter { ... };

Default: —

Context: http

Specifies OTel data export parameters:

  • endpoint — the address of OTLP/gRPC endpoint that will accept telemetry data.
  • interval — the maximum interval between two exports, by default is 5 seconds.
  • batch_size — the maximum number of spans to be sent in one batch per worker, by default is 512.
  • batch_count — the number of pending batches per worker, spans exceeding the limit are dropped, by default is 4.

Example:

otel_exporter {
    endpoint    localhost:4317;
    interval    5s;
    batch_size  512;
    batch_count 4;
}

otel_service_name

Syntax: otel_service_name name;

Default: otel_service_name unknown_service:nginx;

Context: http

Sets the “service.name” attribute of the OTel resource.

otel_trace

Syntax: otel_trace on | off | $variable;

Default: otel_trace off;

Context: http, server, location

Enables or disables OpenTelemetry tracing. The directive can also be enabled by specifying a variable.

Example:

split_clients "$otel_trace_id" $ratio_sampler {
               10%              on;
               *                off;
}

server {
    location / {
        otel_trace         $ratio_sampler;
        otel_trace_context inject;
        proxy_pass         http://backend;
    }
}

otel_trace_context

Syntax: otel_trace_context extract | inject | propagate | ignore;

Default: otel_trace_context ignore;

Context: http, server, location

Specifies how to propagate traceparent/tracestate headers:

  • extract — uses an existing trace context from the request, so that the identifiers of a trace and the parent span are inherited from the incoming request.
  • inject — adds a new context to the request, overwriting existing headers, if any.
  • propagate — updates the existing context (combines extract and inject).
  • ignore — skips context headers processing.

otel_span_name

Syntax: otel_span_name name;

Default: —

Context: http, server, location

Defines the name of the OTel span. By default, it is a name of the location for a request. The name can contain variables.

otel_span_attr

Syntax: otel_span_attr name value;

Default: —

Context: http, server, location

Adds a custom OTel span attribute. The value can contain variables.

Default span attributes

The following span attributes are added automatically:

  • http.method
  • http.target
  • http.route
  • http.scheme
  • http.flavor
  • http.user_agent
  • http.request_content_length
  • http.response_content_length
  • http.status_code
  • net.host.name
  • net.host.port
  • net.sock.peer.addr
  • net.sock.peer.port

Module variables

$otel_trace_id

the identifier of the trace the current span belongs to, for example, 56552bc4daa3bf39c08362527e1dd6c4

$otel_span_id

the identifier of the current span, for example, 4c0b8531ec38ca59

$otel_parent_id

the identifier of the parent span, for example, dc94d281b0f884ea

$otel_parent_sampled

the sampled flag of the parent span, can be 1 or 0

Usage examples

Simple Tracing

Dumping all the requests could be useful even in non-distributed environment.

http {
    otel_trace on;
    server {
        location / {
        proxy_pass http://backend;
        }
    }
}

Parent-based Tracing 

http {
    server {
        location / {
            otel_trace $otel_parent_sampled;
            otel_trace_context propagate;

            proxy_pass http://backend;
        }
    }
}

Ratio-based Tracing 

http {
    # trace 10% of requests
    split_clients $otel_trace_id $ratio_sampler {
        10%     on;
        *       off;
    }

    # or we can trace 10% of user sessions
    split_clients $cookie_sessionid $session_sampler {
        10%     on;
        *       off;
    }

    server {
        location / {
            otel_trace $ratio_sampler;
            otel_trace_context inject;

            proxy_pass http://backend;
        }
    }
}

More Info