Encryption

This document helps you secure NGINX Instance Manager by utilizing certificates.

Warning:
A management server should NEVER be exposed to the public internet. You can mitigate exposure with settings here, but they are not a substitute for preventing unneeded exposure.
Note:
NGINX Plus is provided to be used with NGINX Instance Manager only as a frontend. Using NGINX Plus for other web applications or instances is not permitted.

Prerequisites

  1. Install NGINX Instance Manager Server.
  2. Install NGINX or NGINX Plus.
  3. Start and Enable Instance Manager and NGINX Plus (or NGINX).

Encryption Methods

Use the native SSL encryption methods for NGINX Instance Manager or the recommended approach of using NGINX as an SSL termination reverse proxy.

The following table summarizes the key differences between NGINX Open Source and NGINX Plus (for authentication options with NGINX Instance Manager).

Method no proxy nginx-oss nginx-plus
SSL Termination Included Included Included
Client mTLS Included Included Included
GRPC Health Checks N/A N/A Included

How it works

Instance Manager listens on two ports. The gRPC communication listens only on 127.0.0.1 and is on 10000 by default. The browser interface and API listen on 127.0.0.1 and are on 11000 by default. By using NGINX, we follow our own guidance for any web application. Use NGINX Open Source (“NGINX OSS”) or NGINX Plus for the forward proxy. An instance of NGINX Plus is provided with NGINX Instance Manager. Use it to proxy and secure NGINX Instance Manager.

Common NGINX Snippets

Security Options

These options help prevent and mitigate common attacks that can compromise any web based server. They should be included on all configurations for security reasons. Adjust and use the ones that fit your needs. They are the HSTS settings that should be used with any SSL setup and NGINX. Change the max-age once you’ve verified these settings work for you.

Create a common conf for headers:

# /etc/nginx/http_headers.conf
add_header X-Frame-Options SAMEORIGIN;
# /etc/nginx/https_headers.conf
include http_headers.conf
# add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
# enable the longer max-age above after you test your configuration
add_header Strict-Transport-Security "max-age=86400; includeSubdomains";

# Enable Content Security Policy 
add_header Content-Security-Policy "default-src 'self'; font-src *;img-src * data:; script-src *; style-src *";

# Enable X-XSS protection
add_header X-XSS-Protection "1; mode=block";

# Referrer-Policy
add_header Referrer-Policy "strict-origin";

And use this in the confs for the NGINX proxy:

http {
    include http_headers.conf;

    # Serve HTTP content using the http_headers conf
    server {
        listen 80
    }

    server {
        listen 443 ssl;

        include https_headers.conf

        # This 'location' block inherits the STS header
        location / {
            root /usr/share/nginx/html;
        }

        # Because this 'location' block contains another 'add_header' directive,
        # we must redeclare the STS header
        location /servlet {
            add_header X-Served-By "My Unnecessary Header Example";
            include https_headers.conf
            proxy_pass https://nginx-manager.example.com;
        }
    }
}

Upstreams

The examples below make reference to an upstream group nginx-manager_servers. You can create the same with a conf file similar to the one below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# nginx-manager-upstreams.conf
# Upstreams for NGINX Instance Manager Server API/UI

upstream nginx-manager_servers {
        zone nginx-manager_servers 64k;
        server 127.0.0.1:11000;
        keepalive 64;
}

# vim: syntax=nginx

Status Page

NGINX Plus uses a status page. The following example shows a default configuration on port 8080. Adjust the port to your desired one and add SSL if wanted (or other restrictions like rate limiting).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
# This sample NGINX Plus configuration enables the NGINX Plus API, for live 
# activity monitoring and the built-in dashboard, dynamic configuration of 
# upstream groups, and key-value stores. Keep in mind that any features 
# added to the API in future NGINX Plus releases are enabled 
# automatically by this file.
# Created in May 2018 by NGINX, Inc. for NGINX Plus R14 and later.

# Documentation: 
# https://docs.nginx.com/nginx/admin-guide/monitoring/live-activity-monitoring/
# https://www.nginx.com/blog/live-activity-monitoring-nginx-plus-3-simple-steps

# To conform with the conventional configuration scheme, place this file in 
# the /etc/nginx/conf.d directory and add an 'include' directive that 
# references it in the main configuration file, /etc/nginx/nginx.conf, 
# either by name or with a wildcard expression. Then confirm and reload
# the configuration, for example with this command:
#
#     nginx -t && nginx -s reload

# Additional directives are required in other parts of the 
# configuration:
#
# To collect metrics for an HTTP or TCP/UDP virtual server, you must 
# include the 'status_zone' directive in its 'server' block. See: 
# http://nginx.org/r/status_zone
#
# Similarly, to collect metrics for an upstream server group, you 
# must include the 'zone' directive in the 'upstream' block. See:
# http://nginx.org/r/zone
#
# For more information and instructions, see:
# https://docs.nginx.com/nginx/admin-guide/monitoring/live-activity-monitoring#status_data

# It is recommended to restrict access to the NGINX Plus API so 
# that only authorized users can view metrics and configuration, change 
# configuration, or both. Here are a few options:
#
# (1) Configure your firewall to limit access to port 8080.
#
# (2) Use SSL/TLS client certificates. See:
#    https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/
#
# (3) Enable HTTP Basic authentication (RFC 7617) by uncommenting the 
#    'auth_basic*' directives in the 'server' block below. You can add users 
#    with an htpasswd generator, which is readily available, or reuse an 
#    existing htpasswd file (from an Apache HTTP Server, for example). See: 
#    http://nginx.org/en/docs/http/ngx_http_auth_basic_module.html
#
# (4) Enable access from a defined network and disable it from all others, 
#    by uncommenting the 'allow' and 'deny' directives in the 'server' block
#    below and specifying the appropriate network ranges. See: 
#    http://nginx.org/en/docs/http/ngx_http_access_module.html
#
# You can create further restrictions on write operations, to distinguish
# between users with read permission and those who can change configuration.
# Uncomment the sample 'limit_except' directive in the 'location api' 
# block below. In addition to the HTTP Basic authentication shown, other 
# authentication schemes are supported. See: 
# http://nginx.org/r/limit_except

server {
    # Conventional port for the NGINX Plus API is 8080
    listen 8080;

    access_log off; # Don't log access here (test env)
    
    # Uncomment to use HTTP Basic authentication; see (3) above
    #auth_basic "NGINX Plus API";
    #auth_basic_user_file /etc/nginx/users;

    # Uncomment to use permissions based on IP address; see (4) above
    #allow 10.0.0.0/8;
    #deny all;

    # Conventional location for accessing the NGINX Plus API 
    location /api/ {
        # Enable in read-write mode
        api write=on;

        # Uncomment to further restrict write permissions; see note above
        #limit_except GET {
            #auth_basic "NGINX Plus API";
            #auth_basic_user_file /etc/nginx/admins;
        #}
    }

    # Conventional location of the NGINX Plus dashboard
    location = /dashboard.html {
        root /usr/share/nginx/html;
    }

    # Redirect requests for "/" to "/dashboard.html"
    location / {
        root /usr/share/nginx/html;
        index dashboard.html;
    }

    # Swagger-UI exposure
    location /swagger-ui {
        root /usr/share/nginx/html;
    }

    # Redirect requests for pre-R14 dashboard
    location /status.html {
        return 301 /dashboard.html;
    }
}

# vim: syntax=nginx

Stub Page

NGINX Open Source uses a stub status page. The following example shows a default configuration on port 8080. Adjust the port to your desired one and add SSL if wanted (or other restrictions like rate limiting).

Please Note: You need to compile NGINX Open Source with the –with-http_stub_status_module parameter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# ngx_http_stub_status_module (Available in NGINX F/OSS)
# provides Basic Status information http://nginx.org/en/docs/http/ngx_http_stub_status_module.html

server {
    listen 127.0.0.1:80;
    server_name 127.0.0.1;
    access_log off; # Don't log access here (test env)
    location /nginx_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

# vim: syntax=nginx

SSL Termination

Configure either NGINX Open Source or NGINX Plus for SSL Termination. You can use either of the two methods:

  • native SSL in the Instance Manager config files or
  • use a proxy in the front end to provide SSL Termination

Native SSL Termination

Modify the /etc/nginx-manager/nginx-manager.conf file to include SSL certificates. You will need to verify these certificates are valid and also adjust agent settings to use https.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#
# /etc/nginx-manager/nginx-manager.conf
#

# Configuration file for NGINX Instance Manager Server

# bind address for all service ports (default "127.0.0.1")
bind-address: 0.0.0.0
# gRPC service port for agent communication (default "10000")
grpc-port: 10000
# gRPC-gateway service port for API and UI (default "11000")
gateway-port: 11000

# path to x.509 certificate file (optional)
cert: /etc/ssl/nginx-manager/nginx-manager.crt
# path to x.509 certificate key file (optional)
key: /etc/ssl/nginx-manager/nginx-manager.key

NGINX Proxy SSL Termination

Configure the SSL certificate and key inside the NGINX configuration as you normally would. Refer to the guide NGINX SSL Termination.

SSL Termination for NGINX Plus

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# nginx-manager-noauth.conf
# Proxy UI/API with basic auth to 127.0.0.1 on nginx-manager
# Include the nginx-manager-upstreams.conf for the proxy_pass to work


server {
    # listen    80;
    listen      443 ssl;

    status_zone nginx-manager_noauth_https;
    server_name nginx-manager.example.com;

    # Optional log locations
    # access_log    /var/log/nginx/nginx-manager-noauth-access.log info;
    # error_log     /var/log/nginx/nginx-manager-noauth-error.log;

    # SSL certificates must be valid for the FQDN and placed in the correct directories
    ssl_certificate         /etc/ssl/nginx-manager/nginx-manager.crt;
    ssl_certificate_key     /etc/ssl/nginx-manager/nginx-manager.key;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    location / {
        proxy_pass http://nginx-manager_servers;
        health_check uri=/swagger-ui/;
    }

}

# vim: syntax=nginx

SSL Termination for NGINX OSS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# nginx-manager-noauth.conf
# Proxy UI/API with basic auth to 127.0.0.1 on nginx-manager
# Include the nginx-manager-upstreams.conf for the proxy_pass to work


server {
    # listen    80;
    listen      443 ssl;

    server_name nginx-manager.example.com;

    # Optional log locations
    # access_log    /var/log/nginx/nginx-manager-noauth-access.log info;
    # error_log     /var/log/nginx/nginx-manager-noauth-error.log;

    # SSL certificates must be valid for the FQDN and placed in the correct directories
    ssl_certificate         /etc/ssl/nginx-manager/nginx-manager.crt;
    ssl_certificate_key     /etc/ssl/nginx-manager/nginx-manager.key;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    location / {
        proxy_pass http://nginx-manager_servers;
    }

}

# vim: syntax=nginx

Mutual Client Certificate Auth Setup (mtls)

Using client certificates unique to each endpoint allows you to secure and authorize individual nginx instances with the nginx-manager. You can utilize basic client auth with nginx-manager directly or leverage nginx as a proxy to offload client cert handling or reencrypt traffic back to the nginx-manager. For enterprises, the recommended method is to leverage the included NGINX Plus proxy and configure the options below in the NGINX Plus Section.

Use PKI methods to secure your enterprise. Refer to the following instructions for guidance.

Generate a Certificate Authority (CA) for all the methods described below. This can be on the Instance Manager server for testing. For production, follow your organization’s standards (typically an offline machine without network connections).

The root CA provides a certificate for an intermediate CA which should be secured. The root CA (or intermediate CA) issues client and server certificates. The CA (either root or intermediate) will sign certificate signing requests (CSRs) and issue certificates. The following examples assume you have the following components setup:

  1. CA server (nginx-manager-ca.example.com) with ca.pem bundle
  2. Instance Manager server (nginx-manager.example.com) with key and crt
  3. nginx-agent instances (nginx-1.example.com) with key and crt

Instructions are below for the above steps

  1. Create the CA bundle

Generate the CA key and certificate if not present already. v3 extensions are used for SAN in most examples. We leverage the SAN name and not the CN (due to modern requirements that ignore CN names or give warnings).

Create a configuration file for openssl for the CA containing information similar to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
[req]
default_bits        = 4096
distinguished_name  = req_distinguished_name
prompt              = no
default_md          = sha256
req_extensions      = v3_req

[req_distinguished_name]
countryName                 = US
stateOrProvinceName         = California
localityName                = San Francisco
organizationName            = NGINX, Inc.
commonName                  = nginx-manager_ca

[v3_req]
basicConstraints = CA:true
keyUsage = critical, keyCertSign
subjectAltName = @alt_names

[alt_names]
DNS.1 = nginx-manager-ca.example.com

The next step is to generate a key and crt for the CA, using the cnf file from above, on your CA machine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
echo "Creating CA certificate and key"
openssl req \
    -new \
    -newkey rsa:4096 \
    -days 365 \
    -sha256 \
    -nodes \
    -x509 \
    -keyout ca.key \
    -out ca.pem \
    -config ca.cnf \
    -extensions v3_req

This should produce a ca.key and ca.pem. Do not distribute your ca.key, this must be protected and not shared, it is only to be used by the CA and recommended this machine is secured. Ensure the ca.key is not world readable.

Verify the ca.pem or bundle is valid by running the following command

openssl x509 -in ca.pem -noout -text
  1. Setup Instance Manager Server certificates

On the nginx-manager, create an openssl configuration file paying attention to the SAN names and the IP if needed (especially if you use the default 127.0.0.1). You only need one DNS name for the SAN but if you use a different FQDN for the GUI, place the additional names in the SAN area also. If you plan to proxy 127.0.0.1 directly unencrypted, add that IP also. Call the configuration file server.cnf in the example below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
[req]
prompt             = no
default_bits       = 4096
x509_extensions    = v3_req
req_extensions     = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]
countryName                 = US
stateOrProvinceName         = California
localityName                = San Francisco
organizationName            = NGINX, Inc.
commonName                  = nginx-manager

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = nginx-manager.example.com
IP.1 = 127.0.0.1

Create the key and CSR using the OpenSSL configuration above.

echo "Creating Instance Manager Server key and csr"
openssl req \
    -new \
    -newkey rsa:4096 \
    -nodes \
    -keyout server.key \
    -out server.csr \
    -config server.cnf \
    -extensions v3_req
echo "Verifying the Instance Manager Server csr"
openssl req -in server.csr -noout -verify

You will need to transport the csr to the CA server and run the following on the CA server (not the nginx-manager) to generate the crt we need

echo "Signing Instance Manager Server csr"
openssl x509 \
    -req \
    -days 365 \
    -CA ca.pem \
    -CAkey ca.key \
    -CAcreateserial \
    -in server.csr \
    -out server.crt \
    -extfile server.cnf \
    -extensions v3_req
echo "Verifying Instance Manager Server crt"
openssl verify -CAfile ca.pem server.crt

Copy the server.crt and ca.pem to the Instance Manager Server and have your system trust the ca.pem if desired (follow your OS instructions for doing this.)

sudo mkdir -p /etc/ssl/nginx-manager
sudo mv ca.pem /etc/ssl/nginx-manager
sudo mv server.crt /etc/ssl/nginx-manager/server.crt
sudo mv server.key /etc/ssl/nginx-manager/server.key
sudo chown -R root /etc/ssl/nginx-manager
sudo chmod 640 /etc/ssl/nginx-manager/server.key
  1. Setup the Client certificate and key (repeat for each client)

On the client, we will create a key and a csr to send to the ca.

Note:
About reusing client certificates: While you can add wildcards and multiple SAN names, you should only use a DNS SAN name unique for each client for security reasons. Reusing the client certificate on multiple client lessens security and doesn’t provide authentication unique for each client (since you are sharing certificates)

First generate the client OpenSSL configuration file to use, called nginx1.cnf in our example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[req]
prompt             = no
default_bits       = 2048
x509_extensions    = v3_req
req_extensions     = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]
countryName                 = US
stateOrProvinceName         = California
localityName                = San Francisco
organizationName            = NGINX, Inc.
commonName                  = nginx1

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = nginx-1.example.com

Create the client key and csr using the configuration above.

echo "Creating Agent key and csr"
openssl req \
    -new \
    -newkey rsa:2048 \
    -nodes \
    -keyout agent.key \
    -out nginx1.csr \
    -config nginx1.cnf \
    -extensions v3_req
echo "Verifying the Agent csr"
openssl req -in nginx1.csr -noout -verify

Copy the nginx1.csr and nginx1.cnf to the CA server and run the following commands to sing the csr.

echo "Signing Agent csr"
openssl x509 \
    -req \
    -days 365 \
    -CA ca.pem \
    -CAkey ca.key \
    -CAcreateserial \
    -in nginx1.csr \
    -out nginx1.crt \
    -extfile nginx1.cnf \
    -extensions v3_req
echo "Verifying Agent certificate"
openssl verify -CAfile ca.pem nginx1.crt

Copy the ca.pem and nginx1.crt back to the nginx-1.example.com instance. Issue the OS command to trust the ca.pem for your OS. Place the nginx1.key, nginx1.crt and ca.pem into the /etc/ssl/nginx-manager directory (create this if it doesn’t exist) and call the key and crt agent.key and agent.crt for the following examples.

sudo mkdir -p /etc/ssl/nginx-manager
sudo mv ca.pem /etc/ssl/nginx-manager
sudo mv agent.crt /etc/ssl/nginx-manager/agent.crt
sudo mv agent.key /etc/ssl/nginx-manager/agent.key
sudo chown -R root /etc/ssl/nginx-manager
sudo chmod 640 /etc/ssl/nginx-manager/agent.key

Repeat the above process for each agent you wish to have a unique certificate.

Using Instance Manager for mTLS

Instance Manager provides a way to load certificates for client certificates and mTLS natively. This method is easy to get started with. However, it may be harder to integrate with existing solutions or the provided advanced security options (such as certificate revocation and rotation).

Setup the nginx-manager to utilize the server.crt,key and ca from above. Edit the nginx-manager.conf similar to the below example. The server-name field is needed to match the SAN name we used in the server.crt generation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#
# /etc/nginx-manager/nginx-manager.conf
#

# Configuration file for NGINX Instance Manager Server

# bind address for all service ports (default "localhost")
bind-address: 0.0.0.0
# gRPC service port for agent communication (default "10000")
grpc-port: 10000
# gRPC-gateway service port for API and UI (default "11000")
gateway-port: 11000

# SSL server name for use with cert and key below (optional)
server-name: nginx-manager.example.com
# path to x.509 certificate file (optional)
cert: /etc/ssl/nginx-manager/server.crt
# path to x.509 certificate key file (optional)
key: /etc/ssl/nginx-manager/server.key

# set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
log:
    level: info
    path: /var/log/nginx-manager/
# Metrics default storage path (default "/tmp/metrics") (directory must be already present)
metrics:
    storage-path: /var/nginx-manager/

Restarting the nginx-manager will pickup the encryption options but you also need to update all the nginx-agent endpoints to utilize certificates.

In our nginx1 example instance, modify the nginx-agent.conf file to resemble the example below. Note the tls options we configure. The enable:true indicates we want grpc to be secured. In addition, when we specify the cert and key, we tell the nginx-agent to use client cert authentication with the nginx-manager. Finally, we also add the ca.pem since we generated the certificates off this ca (which must be the same for both the client and server). If the CA was publicly trusted, we could omit the ca option.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#
# /etc/nginx-agent/nginx-agent.conf
#

# Configuration file for NGINX Agent

# specify the server grpc port to connect to
server: nginx-manager.example.com:10000

# tls options
tls:
  # disable tls for testing
  enable: true
  # path to certificate
  cert: /etc/ssl/nginx-manager/agent.crt
  # path to certificate key
  key: /etc/ssl/nginx-manager/agent.key
  # path to CA cert
  ca: /etc/ssl/nginx-manager/ca.pem
log:
  # set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
  level: info
  # set log path. if empty, don't log to file.
  path: /var/log/nginx-agent/
# (optional) tags for this specific instance / machine for inventory purposes
tags:
  # instance:
  location: unspecified
# nginx configuration options
nginx:
  # path of nginx to manage
  bin_path: /usr/sbin/nginx
  # specify stub status URL (see: nginx.org/r/stub_status)
  basic_status_url: "http://localhost:80/nginx_status"
  # specify plus status api url (see nginx.org/r/api)
  plus_api_url: "http://localhost:8080/api"

Restart the agent and you should have gRPC utilizing client authentication. This setup also forces you to use SSL for the UI/API (server-side SSL only).

Leverage NGINX for offloading mTLS

Having NGINX proxy the client certs allows you to offload the encryption tasks which can help with performance and scaling. In this method we can utilize NGINX Plus or NGINX Open Source and will have nginx-manager listen on 127.0.0.1 without encryption, allowing NGINX to listen on the public IP and verify client certificates. You can also combine the method above with this method to secure communications end-to-end.

Configure the nginx-manager to only listen on 127.0.0.1 and use unencrypted communication. This is the default from a fresh installation. Leave the cert and key options commented out. To verify configure the nginx-manager.conf to resemble the following example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#
# /etc/nginx-manager/nginx-manager.conf
#

# Configuration file for NGINX Instance Manager Server

# bind address for all service ports (default "localhost")
bind-address: 127.0.0.1
# gRPC service port for agent communication (default "10000")
grpc-port: 10000
# gRPC-gateway service port for API and UI (default "11000")
gateway-port: 11000

# SSL server name for use with cert and key below (optional)
# server-name: nginx-manager.example.com
# path to x.509 certificate file (optional)
# cert: /etc/ssl/nginx-manager/server.crt
# path to x.509 certificate key file (optional)
# key: /etc/ssl/nginx-manager/server.key

# set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
log:
    level: info
    path: /var/log/nginx-manager/
# Metrics default storage path (default "/tmp/metrics") (directory must be already present)
metrics:
    storage-path: /var/nginx-manager/

Configure the NGINX Plus instance running on the nginx-manager with a grpc conf similar to the following example. Reload nginx (systemctl reload nginx) and now nginx should be listening on 10443 for grpc(s) communication. Note the options to verify client and for the depth in the example below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
# nginx-manager-grpc.conf
# Proxy grpc through tcp 10443 to 127.0.0.1 on nginx-manager
# Can have SSL added (internal name must be on certificate used)
# Replace 10443 with the port you want to use externally

log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'
                                  '"uri":"$uri","http-status":$status,'
                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
                                  '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # We normally expect to receive 
                                           # grpc-status as a trailer
    ''      $sent_http_grpc_status;        # Else use the header, regardless of 
                                           # who generated it
}

server {
    status_zone nginx-manager_grpc;
    listen 10443 http2 ssl;
    server_name nginx-manager.example.com;

    access_log /var/log/nginx/grpc-access.log grpc_json; # Alternate log location and format

    ssl_certificate         /etc/ssl/nginx-manager/nginx-manager.crt;
    ssl_certificate_key     /etc/ssl/nginx-manager/nginx-manager.key;

    ssl_verify_client       on;
    ssl_verify_depth        2;
    ssl_client_certificate  /etc/ssl/nginx-manager/ca.pem;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    location / {
        grpc_pass grpc://nginx-manager_grpc_servers; # grpcs for secure and grpc for insecure
        health_check type=grpc grpc_status=12; # 12=unimplemented
        client_max_body_size 10m;
        client_body_timeout 3000s;
    }

    default_type application/grpc;   # Ensure gRPC for all error responses
    # gRPC-compliant error responses
    # Standard HTTP-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
    #
    error_page 400 = @grpc_internal;
    error_page 401 = @grpc_unauthenticated;
    error_page 403 = @grpc_permission_denied;
    error_page 404 = @grpc_unimplemented;
    error_page 429 = @grpc_unavailable;
    error_page 502 = @grpc_unavailable;
    error_page 503 = @grpc_unavailable;
    error_page 504 = @grpc_unavailable;

    # NGINX-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
    #
    error_page 405 = @grpc_internal; # Method not allowed
    error_page 408 = @grpc_deadline_exceeded; # Request timeout
    error_page 413 = @grpc_resource_exhausted; # Payload too large
    error_page 414 = @grpc_resource_exhausted; # Request URI too large
    error_page 415 = @grpc_internal; # Unsupported media type;
    error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
    error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
    error_page 496 = @grpc_unauthenticated; # Client certificate not presented
    error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
    error_page 500 = @grpc_internal; # Server error
    error_page 501 = @grpc_internal; # Not implemented

    # gRPC error responses
    # Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
    #
    location @grpc_deadline_exceeded {
        add_header grpc-status 4;
        add_header grpc-message 'deadline exceeded';
        return 204;
    }

    location @grpc_permission_denied {
        add_header grpc-status 7;
        add_header grpc-message 'permission denied';
        return 204;
    }

    location @grpc_resource_exhausted {
        add_header grpc-status 8;
        add_header grpc-message 'resource exhausted';
        return 204;
    }

    location @grpc_unimplemented {
        add_header grpc-status 12;
        add_header grpc-message unimplemented;
        return 204;
    }

    location @grpc_internal {
        add_header grpc-status 13;
        add_header grpc-message 'internal error';
        return 204;
    }

    location @grpc_unavailable {
        add_header grpc-status 14;
        add_header grpc-message unavailable;
        return 204;
    }

    location @grpc_unauthenticated {
        add_header grpc-status 16;
        add_header grpc-message unauthenticated;
        return 204;
    }

}

upstream nginx-manager_grpc_servers {
        zone nginx-manager_grpc 64k;
        server 127.0.0.1:10000;
}

# vim: syntax=nginx

On our example nginx-1.example.com instance, edit the nginx-agent.conf to resemble the following example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#
# /etc/nginx-agent/nginx-agent.conf
#

# Configuration file for NGINX Agent

# specify the server grpc port to connect to
server: nginx-manager.example.com:10443

# tls options
tls:
  # disable tls for testing
  enable: true
  # path to certificate
  cert: /etc/ssl/nginx-manager/agent.crt
  # path to certificate key
  key: /etc/ssl/nginx-manager/agent.key
  # path to CA cert
  ca: /etc/ssl/nginx-manager/ca.pem
log:
  # set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
  level: info
  # set log path. if empty, don't log to file.
  path: /var/log/nginx-agent/
# (optional) tags for this specific instance / machine for inventory purposes
tags:
  # instance:
  location: unspecified
# nginx configuration options
nginx:
  # path of nginx to manage
  bin_path: /usr/sbin/nginx
  # specify stub status URL (see: nginx.org/r/stub_status)
  basic_status_url: "http://localhost:80/nginx_status"
  # specify plus status api url (see nginx.org/r/api)
  plus_api_url: "http://localhost:8080/api"

Restart the agent and verify through the logs, or by using the browser interface or API, that we can receive metrics and configuration data. Client cert authentication is now taking place through the NGINX proxy.

Leverage NGINX Plus for advanced mTLS

Using NGINX Plus allows all the features of mTLS but adds the ability to work with other providers, rotate certificates and leverage key-value stores and technologies that can improve performance and security. In this method, NGINX Plus will handle the grpc communications and pass the traffic back internally to nginx-manager. You can pass the traffic back unencrypted also combining the method above.

This is very similar to the method above but we will enable nginx-manager to run securely and modify the nginx proxy to use grpcs instead of grpc.

Edit the nginx-manager.conf to resemble the following example to secure traffic. Note we use 127.0.0.1 instead of 0.0.0.0. This means we only listen on 127.0.0.1 but we use the secure communications path. We leverage the server-name option to match the SAN we used. We could use a separate crt and key also which differs from the proxy crt/key but for now we use the same.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#
# /etc/nginx-manager/nginx-manager.conf
#

# Configuration file for NGINX Instance Manager Server

# bind address for all service ports (default "localhost")
bind-address: 127.0.0.1
# gRPC service port for agent communication (default "10000")
grpc-port: 10000
# gRPC-gateway service port for API and UI (default "11000")
gateway-port: 11000

# SSL server name for use with cert and key below (optional)
server-name: nginx-manager.example.com
# path to x.509 certificate file (optional)
cert: /etc/ssl/nginx-manager/server.crt
# path to x.509 certificate key file (optional)
key: /etc/ssl/nginx-manager/server.key

# set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
log:
    level: info
    path: /var/log/nginx-manager/
# Metrics default storage path (default "/tmp/metrics") (directory must be already present)
metrics:
    storage-path: /var/nginx-manager/

Restarting the nginx-manager now enables secure communication but only internally. Configure the nginx plus instance with a conf file similar to the example below to listen on 10443 for grpcs traffic. Note the proxy option to use grpcs instead of grpc here. You don’t need the grpc crt, key, and ca options here but they’re listed and commented in case you want to leverage multiple certificates that differ on the internal leg.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
# nginx-manager-grpc.conf
# Proxy grpc through tcp 10443 to 127.0.0.1 on nginx-manager
# Can have SSL added (internal name must be on certificate used)
# Replace 10443 with the port you want to use externally

log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'
                                  '"uri":"$uri","http-status":$status,'
                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
                                  '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # We normally expect to receive 
                                           # grpc-status as a trailer
    ''      $sent_http_grpc_status;        # Else use the header, regardless of 
                                           # who generated it
}

server {
    status_zone nginx-manager_grpc;
    listen 10443 http2 ssl;
    server_name nginx-manager.example.com;

    access_log /var/log/nginx/grpc-access.log grpc_json; # Alternate log location and format

    ssl_certificate         /etc/ssl/nginx-manager/nginx-manager.crt;
    ssl_certificate_key     /etc/ssl/nginx-manager/nginx-manager.key;

    ssl_verify_client on;
    ssl_verify_depth 2;
    ssl_client_certificate  /etc/ssl/nginx-manager/ca.pem;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    location / {
        grpc_pass grpcs://nginx-manager_grpc_servers;
        # grpc_ssl_certificate /etc/ssl/nginx-manager/nginx-manager.crt;
        # grpc_ssl_certificate_key /etc/ssl/nginx-manager/nginx-manager.key;
        # grpc_ssl_trusted_certificate /etc/ssl/nginx-manager/ca.pem;
        health_check type=grpc grpc_status=12; # 12=unimplemented
        client_max_body_size 10m;
        client_body_timeout 3000s;
    }

    default_type application/grpc;   # Ensure gRPC for all error responses
    # gRPC-compliant error responses
    # Standard HTTP-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
    #
    error_page 400 = @grpc_internal;
    error_page 401 = @grpc_unauthenticated;
    error_page 403 = @grpc_permission_denied;
    error_page 404 = @grpc_unimplemented;
    error_page 429 = @grpc_unavailable;
    error_page 502 = @grpc_unavailable;
    error_page 503 = @grpc_unavailable;
    error_page 504 = @grpc_unavailable;

    # NGINX-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
    #
    error_page 405 = @grpc_internal; # Method not allowed
    error_page 408 = @grpc_deadline_exceeded; # Request timeout
    error_page 413 = @grpc_resource_exhausted; # Payload too large
    error_page 414 = @grpc_resource_exhausted; # Request URI too large
    error_page 415 = @grpc_internal; # Unsupported media type;
    error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
    error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
    error_page 496 = @grpc_unauthenticated; # Client certificate not presented
    error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
    error_page 500 = @grpc_internal; # Server error
    error_page 501 = @grpc_internal; # Not implemented

    # gRPC error responses
    # Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
    #
    location @grpc_deadline_exceeded {
        add_header grpc-status 4;
        add_header grpc-message 'deadline exceeded';
        return 204;
    }

    location @grpc_permission_denied {
        add_header grpc-status 7;
        add_header grpc-message 'permission denied';
        return 204;
    }

    location @grpc_resource_exhausted {
        add_header grpc-status 8;
        add_header grpc-message 'resource exhausted';
        return 204;
    }

    location @grpc_unimplemented {
        add_header grpc-status 12;
        add_header grpc-message unimplemented;
        return 204;
    }

    location @grpc_internal {
        add_header grpc-status 13;
        add_header grpc-message 'internal error';
        return 204;
    }

    location @grpc_unavailable {
        add_header grpc-status 14;
        add_header grpc-message unavailable;
        return 204;
    }

    location @grpc_unauthenticated {
        add_header grpc-status 16;
        add_header grpc-message unauthenticated;
        return 204;
    }

}

upstream nginx-manager_grpc_servers {
        zone nginx-manager_grpc 64k;
        server 127.0.0.1:10000;
        # server nginx-manager:10000; # Must match SSL cert SN/SAN used for grpcs and for nginx-manager.conf bind-address
}

# vim: syntax=nginx

Reload the NGINX server (systemctl reload nginx) and the new configuration should take effect. Finally, go to your NGINX instance and edit the nginx-agent.conf file to resemble the following example (nginx-1.example.com in our example).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#
# /etc/nginx-agent/nginx-agent.conf
#

# Configuration file for NGINX Agent

# specify the server grpc port to connect to
server: nginx-manager.example.com:10443

# tls options
tls:
  # disable tls for testing
  enable: true
  # path to certificate
  cert: /etc/ssl/nginx-manager/agent.crt
  # path to certificate key
  key: /etc/ssl/nginx-manager/agent.key
  # path to CA cert
  ca: /etc/ssl/nginx-manager/ca.pem
log:
  # set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
  level: info
  # set log path. if empty, don't log to file.
  path: /var/log/nginx-agent/
# (optional) tags for this specific instance / machine for inventory purposes
tags:
  # instance:
  location: unspecified
# nginx configuration options
nginx:
  # path of nginx to manage
  bin_path: /usr/sbin/nginx
  # specify stub status URL (see: nginx.org/r/stub_status)
  basic_status_url: "http://localhost:80/nginx_status"
  # specify plus status api url (see nginx.org/r/api)
  plus_api_url: "http://localhost:8080/api"

grpc Forwarding

NGINX Plus can provide gRPC health checks and forwarding by using a file similar to the one below in the http conf directory (usually conf.d).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
# nginx-manager-grpc.conf
# Proxy grpc through tcp 10002 to 127.0.0.1 on nginx-manager
# Can have SSL added (internal name must be on certificate used)
# Replace 10002 with the port you want to use externally

log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'
                                  '"uri":"$uri","http-status":$status,'
                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
                                  '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # We normally expect to receive 
                                           # grpc-status as a trailer
    ''      $sent_http_grpc_status;        # Else use the header, regardless of 
                                           # who generated it
}

server {
    status_zone nginx-manager_grpc_grpc;
    listen 10443 http2;
    server_name nginx-manager.example.com;

    access_log /var/log/nginx/nginx-manager-grpc-access.log grpc_json; # Alternate log location and format

    location / {
        grpc_pass grpc://nginx-manager_grpc_servers; # Adjust to grpcs for SSL
        health_check type=grpc grpc_status=12; # 12=unimplemented
        client_max_body_size 10m;
        client_body_timeout 3000s;
    }

    default_type application/grpc;   # Ensure gRPC for all error responses
    # gRPC-compliant error responses
    # Standard HTTP-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
    #
    error_page 400 = @grpc_internal;
    error_page 401 = @grpc_unauthenticated;
    error_page 403 = @grpc_permission_denied;
    error_page 404 = @grpc_unimplemented;
    error_page 429 = @grpc_unavailable;
    error_page 502 = @grpc_unavailable;
    error_page 503 = @grpc_unavailable;
    error_page 504 = @grpc_unavailable;

    # NGINX-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
    #
    error_page 405 = @grpc_internal; # Method not allowed
    error_page 408 = @grpc_deadline_exceeded; # Request timeout
    error_page 413 = @grpc_resource_exhausted; # Payload too large
    error_page 414 = @grpc_resource_exhausted; # Request URI too large
    error_page 415 = @grpc_internal; # Unsupported media type;
    error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
    error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
    error_page 496 = @grpc_unauthenticated; # Client certificate not presented
    error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
    error_page 500 = @grpc_internal; # Server error
    error_page 501 = @grpc_internal; # Not implemented

    # gRPC error responses
    # Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
    #
    location @grpc_deadline_exceeded {
        add_header grpc-status 4;
        add_header grpc-message 'deadline exceeded';
        return 204;
    }

    location @grpc_permission_denied {
        add_header grpc-status 7;
        add_header grpc-message 'permission denied';
        return 204;
    }

    location @grpc_resource_exhausted {
        add_header grpc-status 8;
        add_header grpc-message 'resource exhausted';
        return 204;
    }

    location @grpc_unimplemented {
        add_header grpc-status 12;
        add_header grpc-message unimplemented;
        return 204;
    }

    location @grpc_internal {
        add_header grpc-status 13;
        add_header grpc-message 'internal error';
        return 204;
    }

    location @grpc_unavailable {
        add_header grpc-status 14;
        add_header grpc-message unavailable;
        return 204;
    }

    location @grpc_unauthenticated {
        add_header grpc-status 16;
        add_header grpc-message unauthenticated;
        return 204;
    }

}

upstream nginx-manager_grpc_servers {
        zone nginx-manager_grpc 64k;
        server 127.0.0.1:10000;
        # you must use a servername that matches the certificate for SSL Use
}

# vim: syntax=nginx

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
# nginx-manager-grpc-ssl.conf
# Proxy grpc through tcp 10443 with TLS encryption to local servername on nginx-manager
# nginx-agent will use the cert that Plus presents here in the nginx-agent.conf
# Replace 10443 with the port you want to use externally

log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'
                                  '"uri":"$uri","http-status":$status,'
                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
                                  '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # We normally expect to receive 
                                           # grpc-status as a trailer
    ''      $sent_http_grpc_status;        # Else use the header, regardless of 
                                           # who generated it
}

server {
    status_zone         nginx-manager_grpc_grpcs;
    listen              10443 ssl http2;
    server_name         nginx-manager.example.com;

    access_log          /var/log/nginx/nginx-manager-grpc-access.log grpc_json; # Alternate log location and format

    ssl_certificate     /etc/ssl/nginx-manager/public.grpc.crt;   # This cert matches the one in nginx-agent.conf
    ssl_certificate_key /etc/ssl/nginx-manager/public.grpc.key;   

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    location / {
        grpc_pass grpc://nginx-manager_grpc_servers; # Adjust to grpcs for SSL
        health_check type=grpc grpc_status=12; # 12=unimplemented
        client_max_body_size 10m;
        client_body_timeout 3000s;
    }

    default_type application/grpc;   # Ensure gRPC for all error responses
    # gRPC-compliant error responses
    # Standard HTTP-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
    #
    error_page 400 = @grpc_internal;
    error_page 401 = @grpc_unauthenticated;
    error_page 403 = @grpc_permission_denied;
    error_page 404 = @grpc_unimplemented;
    error_page 429 = @grpc_unavailable;
    error_page 502 = @grpc_unavailable;
    error_page 503 = @grpc_unavailable;
    error_page 504 = @grpc_unavailable;

    # NGINX-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
    #
    error_page 405 = @grpc_internal; # Method not allowed
    error_page 408 = @grpc_deadline_exceeded; # Request timeout
    error_page 413 = @grpc_resource_exhausted; # Payload too large
    error_page 414 = @grpc_resource_exhausted; # Request URI too large
    error_page 415 = @grpc_internal; # Unsupported media type;
    error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
    error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
    error_page 496 = @grpc_unauthenticated; # Client certificate not presented
    error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
    error_page 500 = @grpc_internal; # Server error
    error_page 501 = @grpc_internal; # Not implemented

    # gRPC error responses
    # Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
    #
    location @grpc_deadline_exceeded {
        add_header grpc-status 4;
        add_header grpc-message 'deadline exceeded';
        return 204;
    }

    location @grpc_permission_denied {
        add_header grpc-status 7;
        add_header grpc-message 'permission denied';
        return 204;
    }

    location @grpc_resource_exhausted {
        add_header grpc-status 8;
        add_header grpc-message 'resource exhausted';
        return 204;
    }

    location @grpc_unimplemented {
        add_header grpc-status 12;
        add_header grpc-message unimplemented;
        return 204;
    }

    location @grpc_internal {
        add_header grpc-status 13;
        add_header grpc-message 'internal error';
        return 204;
    }

    location @grpc_unavailable {
        add_header grpc-status 14;
        add_header grpc-message unavailable;
        return 204;
    }

    location @grpc_unauthenticated {
        add_header grpc-status 16;
        add_header grpc-message unauthenticated;
        return 204;
    }

}

upstream nginx-manager_grpc_servers {
        zone nginx-manager_grpc 64k;
        server 127.0.0.1:10000;
}

# vim: syntax=nginx

To encrypt the internal traffic on the server (only needed if you plan to expose externally or proxy from another instance or require this), you need to add the certificates to the nginx-manager configuration file. Add the certificates and restart the service to enable grpc encryption locally.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#
# /etc/nginx-manager/nginx-manager.conf
#

# Configuration file for NGINX Instance Manager Server

# bind address for all service ports (default "127.0.0.1")
bind-address: nginx-manager
# gRPC service port for agent communication (default "10000")
grpc-port: 10000
# gRPC-gateway service port for API and UI (default "11000")
gateway-port: 11000

# path to x.509 certificate file (optional) - must match bind-address in SAN or CN
cert: /etc/ssl/nginx-manager/nginx-manager.crt
# path to x.509 certificate key file (optional)
key: /etc/ssl/nginx-manager/nginx-manager.key

# set log level (panic, fatal, error, info, debug, trace; default: info) (default "info")
log:
    level: info
    path: /var/log/nginx-manager/
# Metrics default storage path (default "/tmp/metrics") (directory must be already present)
metrics:
    storage-path: /var/nginx-manager/
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
# nginx-manager-grpcs-ssl.conf
# Proxy grpc through tcp 10443 with TLS encryption to local servername on nginx-manager
# nginx-agent will use the cert that Plus presents here in the nginx-agent.conf
# Replace 10443 with the port you want to use externally

log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'
                                  '"uri":"$uri","http-status":$status,'
                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr"'
                                  '"rx-bytes":$request_length,"tx-bytes":$bytes_sent}';

map $upstream_trailer_grpc_status $grpc_status {
    default $upstream_trailer_grpc_status; # We normally expect to receive 
                                           # grpc-status as a trailer
    ''      $sent_http_grpc_status;        # Else use the header, regardless of 
                                           # who generated it
}

server {
    status_zone         nginx-manager_grpcs_grpcs;
    listen              10443 ssl http2;
    server_name         nginx-manager.example.com;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 24h;
    ssl_session_tickets off;

    ssl_protocols   TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers   off;

    access_log          /var/log/nginx/nginx-manager-grpc-access.log grpc_json;   # Alternate log location and format

    ssl_certificate     /etc/ssl/nginx-manager/public.grpc.crt;   # This cert matches the one in nginx-agent.conf
    ssl_certificate_key /etc/ssl/nginx-manager/public.grpc.key;   

    location / {
        grpc_pass                       grpcs://nginx-manager_grpc_servers;   # Adjust to grpcs for SSL
        grpc_ssl_certificate            /etc/ssl/nginx-manager/nginx-manager.crt;   # This cert matches the one in nginx-manager.conf
        grpc_ssl_certificate_key        /etc/ssl/nginx-manager/nginx-manager.key;
        health_check        type=grpc   grpc_status=12;                 # 12=unimplemented
        client_max_body_size 10m;
        client_body_timeout 3000s;
    }

    default_type application/grpc;   # Ensure gRPC for all error responses
    # gRPC-compliant error responses
    # Standard HTTP-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md
    #
    error_page 400 = @grpc_internal;
    error_page 401 = @grpc_unauthenticated;
    error_page 403 = @grpc_permission_denied;
    error_page 404 = @grpc_unimplemented;
    error_page 429 = @grpc_unavailable;
    error_page 502 = @grpc_unavailable;
    error_page 503 = @grpc_unavailable;
    error_page 504 = @grpc_unavailable;

    # NGINX-to-gRPC status code mappings
    # Ref: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
    #
    error_page 405 = @grpc_internal; # Method not allowed
    error_page 408 = @grpc_deadline_exceeded; # Request timeout
    error_page 413 = @grpc_resource_exhausted; # Payload too large
    error_page 414 = @grpc_resource_exhausted; # Request URI too large
    error_page 415 = @grpc_internal; # Unsupported media type;
    error_page 426 = @grpc_internal; # HTTP request was sent to HTTPS port
    error_page 495 = @grpc_unauthenticated; # Client certificate authentication error
    error_page 496 = @grpc_unauthenticated; # Client certificate not presented
    error_page 497 = @grpc_internal; # HTTP request was sent to mutual TLS port
    error_page 500 = @grpc_internal; # Server error
    error_page 501 = @grpc_internal; # Not implemented

    # gRPC error responses
    # Ref: https://github.com/grpc/grpc-go/blob/master/codes/codes.go
    #
    location @grpc_deadline_exceeded {
        add_header grpc-status 4;
        add_header grpc-message 'deadline exceeded';
        return 204;
    }

    location @grpc_permission_denied {
        add_header grpc-status 7;
        add_header grpc-message 'permission denied';
        return 204;
    }

    location @grpc_resource_exhausted {
        add_header grpc-status 8;
        add_header grpc-message 'resource exhausted';
        return 204;
    }

    location @grpc_unimplemented {
        add_header grpc-status 12;
        add_header grpc-message unimplemented;
        return 204;
    }

    location @grpc_internal {
        add_header grpc-status 13;
        add_header grpc-message 'internal error';
        return 204;
    }

    location @grpc_unavailable {
        add_header grpc-status 14;
        add_header grpc-message unavailable;
        return 204;
    }

    location @grpc_unauthenticated {
        add_header grpc-status 16;
        add_header grpc-message unauthenticated;
        return 204;
    }

}

upstream nginx-manager_grpc_servers {
        zone nginx-manager_grpc 64k;
        server nginx-manager:10000; # note the servername above must be on the cert /etc/ssl/nginx-manager/nginx-manager.crt;
}

# vim: syntax=nginx