Prometheus-njs

Expose Prometheus metrics endpoint directly from NGINX Plus.

Module Info

The nginx-plus-module-prometheus module is an njs module that converts miscellaneous NGINX Plus status metrics exposed by the API module to a Prometheus compliant format. The module uses subrequests to the /api endpoint to access the metrics. In case you have configured dynamic upstream routing with generic names for upstream groups, the module can understand replacements for these names and display the correct statistics.

Exported Metrics

The following NGINX Plus status metrics are exported to Prometheus:

Note:

The state metric values in /http/upstreams/ and /stream/upstreams/ are converted using the following rule:

NGINX Prometheus
“up” 1
“draining” 2
“down” 3
“unavail” 4
“checking” 5
“unhealthy” 6

Installation

Install the nginx-plus-module-prometheus module.

  • For Amazon Linux 2, CentOS, Oracle Linux, and RHEL:

    yum install nginx-plus-module-prometheus
    
  • For Amazon Linux 2023, AlmaLinux, Rocky Linux:

    dnf install nginx-plus-module-prometheus
    
  • For Debian and Ubuntu:

    apt-get install nginx-plus-module-prometheus
    
  • For SLES:

    zypper install nginx-plus-module-prometheus
    
  • For Alpine:

    apk add nginx-plus-module-prometheus
    

    For FreeBSD:

    pkg install nginx-plus-module-prometheus
    
Note:
The nginx-plus-module-njs module will also be installed together with the module.

Configuration

After module installation, perform the following steps in NGINX Plus configuration file (nginx.conf):

  1. Enable the nginx-plus-module-njs module in the top‑level context:

    load_module modules/ngx_http_js_module.so;
    
    http {
        # ...
    }
    
  2. Include the prometheus.js file:

    http {
        # ...
        js_import /usr/share/nginx-plus-module-prometheus/prometheus.js;
    }
    
  3. Create a location for Prometheus metrics, for example, /metrics:

    location = /metrics {
        js_content prometheus.metrics;
    }
    
  4. Enable the API to be able to expose the /metrics endpoint from Prometheus:

    location /api {
        api;
        #...
    }
    
  5. (optional, in case of “too big subrequest response” error in error log) Since the module uses subrequests for API calls, you may need to increase the size of the buffer that stores the response body of a subrequest:

    http {
        # ...
        subrequest_output_buffer_size 32k;
    }
    
  6. Configure Prometheus to obtain metrics from NGINX Plus by specifying the network address of the NGINX Plus instance in a scrape_config section of the Prometheus configuration file.

Upgrade Instructions

When upgrading Prometheus-njs version to version 1.3.1 and later, it is important to update NGINX Plus configuration file with the changes introduced in NGINX Plus R22 and njs 0.4.0:

See Configuration for the example of Prometheus-njs configuration in NGINX Plus configuration file.

Module Variables

The module supports several embedded variables:

The variables can be set in NGINX Plus configuration file with the set directive.

Using the $prom_keyval Variable

The $prom_keyval variable has been created to correctly show statistics for dynamic configuration of upstream routing: when names of upstreams are generic and these names are dynamically replaced by real names from the key-value storage.

To add the $prom_keyval variable, add the set directive to the location that exposes metrics to Prometheus (for example, = /metrics), in the following format:

set $prom_keyval "upstream_keyval";

where $prom_keyval will hold all values from upstream_keyval key-value storage specified in the keyval_zone directive:

http {

    #...

    keyval_zone zone=upstream_keyval:64k;
    keyval      $domain $upstream zone=http_upstream_keyval;

    upstream 0 {
        zone   http_backend 64k;
        server backend1.example.com;
        server backend2.example.com;
    }

    server {
        #...

        location / {
            proxy_pass http://$upstream;
            #...
        }

        location /api {
            api;
            #...
        }

        location = /metrics {
            set        $prom_keyval "http_upstream_keyval";
            js_content prometheus.metrics;
        }
    }
}

Using the $prom_keyval_stream Variable

In order to perform dynamic replacements in the stream {} context, you will need to specify a new key-value storage in the stream block and set the name of the key-value storage in the $prom_keyval_stream variable:


http {

    # the contents of the 'http' block is the same as for the $prom_keyval example
    # except the 'location /metrics' block:
    # ...

        location = /metrics {
            set $prom_keyval        "http_upstream_keyval";
            set $prom_keyval_stream "stream_upstream_keyval";
            js_content              prometheus.metrics;
        }
    }
}

stream {

    keyval_zone zone=stream_keyval:32k;

    upstream stream_backend {
        zone   stream_backend 64k;
        server backend1.example.com:12345;
        server backend2.example.com:12345;
    }

    server {
        listen 12345;

        proxy_pass  stream_backend;
        status_zone backend_stream_zone;
    }
}

Using the $prom_metrics_disabled Variable

The $prom_metrics_disabled variable disables exporting particular NGINX Plus status metrics to Prometheus. If you are concerned about the amount of storage NGINX Plus metrics are taking in Prometheus, you can disable particular endpoints via this variable. The variable value must be a comma separated list of NGINX Plus endpoints:

#...
location /metrics {
    set $prom_keyval           "http_upstream_keyval";
    set $prom_keyval_stream    "stream_upstream_keyval";
    set $prom_metrics_disabled "stream/upstreams, resolvers";
    js_content                 prometheus.metrics;
}
#...

You can disable exporting the following NGINX Plus status metrics to Prometheus:

  • nginx
  • processes
  • resolvers
  • connections
  • ssl
  • slabs
  • workers
  • http/requests
  • http/caches
  • http/upstreams
  • http/server_zones
  • http/location_zones
  • http/limit_conns
  • http/limit_reqs
  • stream/upstreams
  • stream/server_zones
  • stream/zone_sync
  • stream/limit_conns

Example

load_module modules/ngx_http_js_module.so;

#...

http {

    js_import /usr/share/nginx-plus-module-prometheus/prometheus.js;

    subrequest_output_buffer_size 32k;

    upstream backend {
        zone   backend 64k;
        server backend1.example.com;
        server backend2.example.com;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend;
        }

        location /api {
            api;
        }

        location = /metrics {
            set $prom_keyval           "http_upstream_keyval";
            set $prom_keyval_stream    "stream_upstream_keyval";
            set $prom_metrics_disabled "stream/upstreams, resolvers";
            js_content                 prometheus.metrics;
        }

        status_zone backend_zone;
    }
}

stream {

    keyval_zone zone=stream_keyval:32k;

    upstream stream_backend {
        zone   stream_backend 64k;
        server backend1.example.com:12345;
        server backend2.example.com:12345;
    }

    server {
        listen 12345;

        proxy_pass  stream_backend;
        status_zone backend_stream_zone;
}

More Info